For around 100 hosts to monitor use 2 cores, 2GB RAM and 40G space.

Install CentOS 8 minimal, enable NTPD, set static IP (use your internal DNS servers if wanting to resolve by internal hostnames.. BUT in /etc/resolv.conf add some public DNS servers after your internal DNS servers, this is so if your internal DNS is down Zabbix can still send an email out using DNS resolution from the public servers) and a fqdn that you will add to your DNS system later. If using multiple NICS to monitor multiple networks, add a gateway only to the NIC that can access other networks (this is typically not your management LAN or other isolated networks)

1. Reboot and perform a yum update + install vim then reboot again…

dnf update && dnf install vim
shutdown -r now

Create user with password

useradd sudo_username && passwd sudo_username

Add user to wheel group for sudo privileges

usermod -aG wheel sudo_username

Log out of root and into newly created account.

Disable root login over SSH

sudo vim /etc/ssh/sshd_config

Add line

PermitRootLogin no

Restart sshd

sudo systemctl restart sshd

Configure your hosts file with your fqdn

sudo vim /etc/hosts

Add something like:

10.254.157.147 zabbix.domain.com zabbix 

sudo firewall-cmd --permanent --zone=public --add-port=80/tcp
sudo firewall-cmd --permanent --zone=public --add-port=443/tcp
sudo firewall-cmd --permanent --zone=public --add-port=10051/tcp
sudo firewall-cmd --reload
sudo systemctl daemon-reload

This will help prevent the baddies from brute forcing your SSH password…

Install epel repo, install fail2ban and enable it

sudo dnf install epel-release
sudo dnf install fail2ban
sudo systemctl enable fail2ban

Create a jail for sshd

sudo vim /etc/fail2ban/jail.d/sshd.local

Add as follows:

[sshd]
enabled = true
mode   = aggressive
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s

[selinux-ssh]
enabled = true
port     = ssh
logpath  = %(auditd_log)s

Restart fail2ban

sudo systemctl restart fail2ban

sudo dnf install wget mariadb-server httpd policycoreutils-python-utils

sudo systemctl enable mariadb
sudo systemctl start mariadb

Secure the MariaDB install

sudo mysql_secure_installation

sudo dnf install epel-release
wget http://rpms.remirepo.net/enterprise/remi-release-8.rpm
sudo rpm -Uvh remi-release-8.rpm
sudo dnf module enable php:remi-7.4
sudo dnf install php

Search for available modules

dnf search php | grep php74

Install modules

sudo dnf install php-gd php-bcmath php-ctype php-xml php-xmlreader php-xmlwriter php-session php-mysqlnd php-mbstring php-gettext php-ldap php-mysqli php-pecl-mysql

Edit php.ini to set some variables…

sudo vim /etc/php.ini

Set the following variables as such

post_max_size = 16M
max_execution_time = 300
max_input_time = 300
date.timezone = America/Los_Angeles

ofc use your own time zone…

https://www.zabbix.com/download?zabbix=5.0&os_distribution=centos&os_version=8&db=mysql&ws=apache

a. Install Zabbix repository documentation

sudo rpm -Uvh https://repo.zabbix.com/zabbix/5.0/rhel/8/x86_64/zabbix-release-5.0-1.el8.noarch.rpm
sudo dnf clean all

b. Install Zabbix server, frontend, agent

sudo dnf install zabbix-server-mysql zabbix-web-mysql zabbix-apache-conf zabbix-agent

c. Create initial database documentation

sudo mysql -uroot -p
###password###
mysql> create database zabbix character set utf8 collate utf8_bin;
mysql> create user zabbix@localhost identified by 'password';
mysql> grant all privileges on zabbix.* to zabbix@localhost;
mysql> quit;
sudo zcat /usr/share/doc/zabbix-server-mysql*/create.sql.gz | mysql -uzabbix -p zabbix

d. Configure the database for Zabbix server

sudo vim /etc/zabbix/zabbix_server.conf

Enter you MariaDB password and set pingers to 100

DBPassword=password
StartPingers=100

sudo vim /etc/php-fpm.d/zabbix.conf

Set the timezone similar to the following:

php_value date.timezone America/Los_Angeles

https://www.zabbix.com/documentation/current/manual/installation/install_from_packages/rhel_centos
https://bestmonitoringtools.com/how-to-install-zabbix-server-on-centos-or-rhel/
https://support.zabbix.com/browse/ZBX-14626

We are going to set some rules but also temporarily put SELinux in Permissive mode. This is needed because until you setup Zabbix agents, SNMP queries/etc as there won't be any “deny” entries in selinux's audit log to create custom rules from. Sooo…. leave selinux in permissive mode until you have everything setup and working (for a day or a week).. then use audi2allow.

Temporarily set selinux to permissive

sudo setenforce 0

sudo setsebool -P httpd_can_connect_zabbix on
sudo setsebool -P zabbix_can_network on
service httpd restart

(do this after you have the server up and hosts/agents/etc configured and communicating)

sudo grep "denied.*zabbix_server" /var/log/audit/audit.log | audit2allow -M zabbix_server
sudo semodule -i zabbix_server.pp
sudo grep "denied.*zabbix_script" /var/log/audit/audit.log | audit2allow -M zabbix_script
sudo semodule -i zabbix_script.pp
sudo setenforce 1

Start Zabbix server and agent processes and make it start at system boot:

sudo systemctl restart zabbix-server zabbix-agent httpd
sudo systemctl enable zabbix-server zabbix-agent httpd

Now your Zabbix server is up and running!

Connect to your newly installed Zabbix frontend: http://server_ip_or_name/zabbix

Go through step, the default username is Admin (case sensitive) and the password is zabbix.

https://www.zabbix.com/documentation/5.0/manual/installation/requirements/best_practices

This section contains best practices that should be observed in order to set up Zabbix in a secure way.

The practices contained here are not required for the functioning of Zabbix. They are recommended for better security of the system.

The principle of least privilege should be used at all times for Zabbix. This principle means that user accounts (in Zabbix frontend) or process user (for Zabbix server/proxy or agent) have only those privileges that are essential to perform intended functions. In other words, user accounts at all times should run with as few privileges as possible.

Giving extra permissions to 'zabbix' user will allow it to access configuration files and execute operations that can compromise the overall security of infrastructure. When implementing the least privilege principle for user accounts, Zabbix frontend user types should be taken into account. It is important to understand that while a “Zabbix Admin” user type has less privileges than “Zabbix Super Admin” user type, it has administrative permissions that allow managing configuration and execute custom scripts.

Some information is available even for non-privileged users. For example, while Administration → Scripts is not available for non-Super Admins, scripts themselves are available for retrieval by using Zabbix API. Limiting script permissions and not adding sensitive information (like access credentials, etc) should be used to avoid exposure of sensitive information available in global scripts.

In the default configuration, Zabbix server and Zabbix agent processes share one 'zabbix' user. If you wish to make sure that the agent cannot access sensitive details in server configuration (e.g. database login information), the agent should be run as a different user:

sudo useradd zabbix-agent && sudo passwd zabbix-agent
sudo mkdir /run/zabbix-agent
sudo chown zabbix-agent:zabbix-agent /run/zabbix-agent
sudo chown zabbix-agent:zabbix-agent /var/log/zabbix/zabbix_agentd.log
sudo vim /etc/zabbix/zabbix_agentd.conf

Find:

### Option: User
#       Drop privileges to a specific, existing user on the system.
#       Only has effect if run as 'root' and AllowRoot is disabled.
#
# Mandatory: no
# Default:
# User=zabbix

Uncomment User=zabbix and change/add as follows:

PidFile=/run/zabbix-agent/zabbix_agentd.pid
User=zabbix-agent

Edit the systemd file…

sudo systemctl edit --full zabbix-agent.service

Modify the following as follows:

PIDFile=/run/zabbix-agent/zabbix_agentd.pid
RuntimeDirectory=zabbix-agent
User=zabbix-agent
Group=zabbix-agent

sudo vim /etc/logrotate.d/zabbix-agent

Change zabbix to zabbix-agent on the following line

        create 0664 zabbix-agent zabbix-agent

Reload systemd and Restart the agent

sudo systemctl daemon-reload
sudo systemctl restart zabbix-agent

Check that the agent is running as the new user:

sudo ps aux | grep zabbix_agentd

The username in the first column should show something like:

zabbix-+

It should not show:

zabbix

UTF-8 is the only encoding supported by Zabbix. It is known to work without any security flaws. Users should be aware that there are known security issues if using some of the other encodings.

On RHEL/Centos, install mod_ssl package:

sudo dnf install mod_ssl

Create directory for SSL keys:

sudo mkdir -p /etc/httpd/ssl/private
sudo chmod 700 /etc/httpd/ssl/private

Create SSL certificate (increase days if using internal CA and optionally rsa to 4096):

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/httpd/ssl/private/apache-selfsigned.key -out /etc/httpd/ssl/apache-selfsigned.crt

Fill out the prompts appropriately. The most important line is the one that requests the Common Name. You need to enter the domain name that you want to be associated with your server. You can enter the public IP address instead if you do not have a domain name. We will use example.com in this article.

Example:

Country Name (2 letter code) [XX]:
State or Province Name (full name) []:
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:example.com
Email Address []:

Edit Apache SSL configuration:

sudo vim /etc/httpd/conf.d/ssl.conf

Modify similar to the following:

DocumentRoot "/usr/share/zabbix"
ServerName example.com:443
SSLCertificateFile /etc/httpd/ssl/apache-selfsigned.crt
SSLCertificateKeyFile /etc/httpd/ssl/private/apache-selfsigned.key

Restart the Apache service to apply the changes:

sudo systemctl restart httpd.service

Add a virtual host to Apache configuration and set permanent redirect for document root to Zabbix SSL URL. Do not forget to replace example.com with the actual name of the server.

sudo vim /etc/httpd/conf/httpd.conf

Add the lines:
<code>
<VirtualHost *:*>
    ServerName example.com
    Redirect permanent / https://example.com
</VirtualHost>

Restart the Apache service to apply the changes:

sudo systemctl restart httpd.service

To protect Zabbix frontend against protocol downgrade attacks, we recommend to enable HSTS policy on the web server.

For example, to enable HSTS policy for your Zabbix frontend in Apache configuration:

sudo vim /etc/httpd/conf/httpd.conf

add the following directive to your virtual host's configuration:

<VirtualHost *:443>
   Header set Strict-Transport-Security "max-age=31536000"
</VirtualHost>

Restart the Apache service to apply the changes:

sudo systemctl restart httpd.service

To protect Zabbix frontend against Cross Site Scripting (XSS), data injection, and some other attacks of a similar type, we recommend enabling Content Security Policy on the web server. To do so, you need to configure the web server to return the Content-Security-Policy HTTP header.

To enable CSP for your Zabbix frontend in Apache configuration, edit:

sudo vim /etc/httpd/conf/httpd.conf

For example, if it is planned that all content will come from the site's origin (excluding subdomains), you can add the following directive to your virtual host's configuration:

<VirtualHost *:*>
 Header set Content-Security-Policy "default-src 'self';"
</VirtualHost>

Restart the Apache service to apply the changes:

sudo systemctl restart httpd.service

It is recommended to disable all web server signatures as part of the web server hardening process. The web server is exposing software signature by default:

The signature can be disabled by adding two lines to the Apache (used as an example) configuration file:

  ServerSignature Off
  ServerTokens Prod

PHP signature (X-Powered-By HTTP header) can be disabled by changing the php.ini configuration file (signature is disabled by default):

  expose_php = Off

Web server restart is required for configuration file changes to be applied.

Additional security level can be achieved by using the mod_security (package libapache2-mod-security2) with Apache. mod_security allows to remove server signature instead of only removing version from server signature. Signature can be altered to any value by changing “SecServerSignature” to any desired value after installing mod_security.

Please refer to documentation of your web server to find help on how to remove/change software signatures.

It is recommended to disable default error pages to avoid information exposure. Web server is using built-in error pages by default:

Default error pages should be replaced/removed as part of the web server hardening process. The “ErrorDocument” directive can be used to define a custom error page/text for Apache web server (used as an example).

Please refer to documentation of your web server to find help on how to replace/remove default error pages.

It is recommended to remove the web server test page to avoid information exposure. By default, web server webroot contains a test page called index.html (Apache2 on Ubuntu is used as an example):

The test page should be removed or should be made unavailable as part of the web server hardening process.

• Make groups for each set of devices that you use where one group needs to be monitored or treated differently than another
• Make groups representing security groups where you'll place togther similar devices in the same security context; this will come in handy for devices that have Zabbix agents so you can specify different PSKs per group and update then as a group.

• Active agents are more secure for the monitored devices since the agents send data to the Zabbix server instead of the server being able to send to the agents. This is useful for over the Internet if no Proxy was setup. Active agents are also less resource intensive for both the Zabbix server and the monitored host. In the event of a network interruption an active agent will send data once the connection is restored where with a passive agent the data is lost. The main negative to an active agent is that you can't run scripts from the Zabbix server to the monitored host. When using an active agent ZBX availability won't show in the “Availability” column. You should probably also add the “ping” template in addition to any OS template so you can see if the host is up.
• Passive agents allow you to run commands on remote hosts and there are some queuing benefits.
• I'm going to default to active agents for security and resource reasons.
• When you setup a host you specify whether the monitor is active or or not (not=passive).

• Download the template: https://share.zabbix.com/printers/universal-printer-template
• In the Web UI import the template: Configuration → Templates → Import → Accept Defaults
• Create Host Group: Configuration → Host groups → Create host group → create a name such as Printers…
• Find a printer, get the IP and add as a host → Configuration → Hosts → Create host (Host name=Printer Share Name, Groups=Printers, Add SNMP connection and enter IP, use SNMPv2 if possible, set the SNMP community, link the “Template Printer Universal SNMPv2 template, save.
• For more info select a specific template for your device or edit the template with the correct OID from the manufacturer MIB., you'll probably need to edit the OIDs anyway…

In this case we'll use printers since we are on the subject.

• Configuration → Discovery → Create Discovery Rule → Name it something specific to the subnets searched, snmp and the OID info you're looking for; Checks should be SNMPv1/v2 with a community name of public and some OID that you want to target your desired printers.

https://bestmonitoringtools.com/zabbix-alerts-setup-zabbix-email-notifications-escalations/

• Administration → Media Types → Email → Enter info then Update → Test
• Optionally modify the Email Message Templates to include the company/zabbix server info in the subject/body. Message Templates are found in Administration → Media Types → Email → Message Templates
• Configuration → Actions → Create action → Label it “Email Something…”; Set User or User Group; Send only to Email → Add → Add.
• Administration → Users → Select User → Media → Add → Type: Email; Send to: someone@somewhere.lom; Optionally set severity levels and service times → Add → Update

https://www.zabbix.com/documentation/current/manual/encryption/using_pre_shared_keys

Generate a PSK (Since we are using PSK I'd recommend creating a PSK per security group, for example domain controllers would be 1 group, general servers another and workstations another; individual PSKs would be ideal but if I'm going to do that right then I may as well just use certificates and today isn't a good day to tackle that as I'm barely able to remember what I just read… this sentence is a testament to that.))..?

Install Agent

sudo rpm -Uvh https://repo.zabbix.com/zabbix/5.2/rhel/8/x86_64/zabbix-release-5.2-1.el8.noarch.rpm
sudo dnf install zabbix-agent
sudo systemctl enable zabbix-agent
sudo systemctl restart zabbix-agent

Change the path to zabbix-agent and user to zabbix-agent if you changed the user running the agent from zabbix to zabbix-agent

sudo mkdir -p /home/zabbix/PSK
sudo openssl rand -hex 32 > /home/zabbix/PSK/PSK-ID.psk (the PSK-ID should a descriptive name of the security group it represents)
sudo chown -R zabbix:zabbix /home/zabbix
sudo vim /etc/zabbix/zabbix_agentd.conf

Modify the following

TLSConnect=psk
TLSAccept=psk
TLSPSKFile=/home/zabbix/PSK/PSK-ID.psk
TLSPSKIdentity=PSK-ID
Server=10.1.2.3 #(IP or servername of the Zabbix server)
ServerActive=10.1.2.3 #(IP or servername of the Zabbix server)
Hostname=host.domain.com #(Name of the host that zabbix agent is running on, this host...)

Fix Selinux issues http://www.ptltech.com/wp-content/uploads/2017/08/Zabbix_Agent_Install_CentOS7.pdf

sudo setenforce 0
sudo systemctl restart zabbix-agent
sudo grep "denied.*zabbix_agentd.*zabbix_agent_t" /var/log/audit/audit.log | audit2allow -M zabbix_agentd_zabbix_agent_t
sudo semodule -i zabbix_agentd_zabbix_agent_t.pp
sudo setenforce 1
sudo systemctl restart zabbix-agent

Enter the PSK ID and key when prompted during the install. If using active communication put the zabbix fqdn/ip in the “Server or Proxy for active checks:” field.

Configuration → Hosts → Select all the hosts of 1 security group where a Zabbix agent is installed → Mass update → Encryption → Connections → Uncheck “No encryption” and check “PSK”, → PSK and enter the “PSK identity” together with the associated “PSK” → Update

I had an issue where to server refused to stay up after adding some SNMP hosts. The logs showed the error “please increase CacheSize configuration parameter”. This is what I did to fix it.

sudo vim /etc/zabbix/zabbix_server.conf

Find and modify the following line (it was defaulted to 8MB)

CacheSize=128M

I was getting this notice often “Zabbix LLD worker processes more than 75% busy Host: Zabbix server”. The fix was change the default value of StartDiscoverers=1 to:

StartDiscoverers=10

Enable Housekeeping (uncomment these entries)

HousekeepingFrequency=1

MaxHousekeeperDelete=5000

https://www.zabbix.com/forum/zabbix-help/381106-disabling-email-alert-for-specific-trigger

Go to Monitoring -> Problems -> click on problem then Configuration -> Click on Tags (if this is greyed out then go back and click on the parent item, like "Discovered by....") then click on Tags again -> add something like "do-not-email" with a value of 1.

Now got to Configuration -> Actions and select your E-Mail action. Add a condition, tag name, does not equal.

Next, test, hopefully this will do it.. I haven't tested it yet... update, it seems to be working!

Also, when in this configuration you can change the Severity of a Trigger, so for Ethernet interface ups and downs you can set it to informational if you don't want all these "problem" alerts.

For decreasing alerts you can increase the duration interval in the triggers, just keep going to the parent trigger until it's editable (or stick close to the device being monitored if you want different variables on that device only): https://serverfault.com/questions/645760/optimal-configuration-of-processor-load-is-too-high-trigger-in-zabbix