Tag Archives: Handy Tips

Zabbix and the Docker API, Part 3: Control

2026-05-06 Janis Eidaks

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/zabbix-and-the-docker-api-part-3-control/32961/

In this blog post, you will learn how to add a simple container remote control capability to Zabbix in order to start, stop, or restart containers from within the discovered host.

You might be wondering, why spend the effort to create a host for each template? Well, that’s because we define a manual script to control the container from within the Zabbix frontend. That’s neat, right? And why stop there? We can also implement a trigger action that automatically restarts the container if it crashes for any reason.

Zabbix server configuration changes

First, we will require global script execution in your Zabbix server configuration:

# nano /etc/zabbix/zabbix_server.conf
EnableGlobalScripts=1
# systemctl restart zabbix-server

Script configuration in frontend

We can create a script in the section Alerts > Scripts. In the script, fill out the specified parameters shown below – the scope, type, and command. Then specify to which hosts this command will apply, as well as the user group that will be able to execute this. This script will take advantage of the user macros and built-in macros to fill the required info in the command to make a correct post request.

● Script
▪ Name: Container action
▪ Scope : Manual host action
▪ Type: Script
▪ Execute on: Zabbix server
▪ Commands: curl -sS -X POST https://{$DOCKER.IP}:{$DOCKER.PORT}/containers{HOST.NAME}/{MANUALINPUT} --cert /etc/zabbix/ssl/certs/client-cert.pem --key /etc/zabbix/ssl/keys/client-key.pem --cacert /etc/zabbix/ssl/ca/ca.pem
▪ Description: Manual action to restart,stop,start container
▪ Host group: Selected: Docker
▪ User group: Zabbix administrators 
▪ Req host perm: Write

● Advanced configuration
▪ Enable user input Check
▪ Input prompt Specify action for container {HOST.NAME}:
▪ Input type: Dropdown 
▪ Dropdown options: restart,stop,start
▪ Enable Confirmation: Check
▪ Confirmation text: Confirm to {MANUALINPUT} container: {HOST.NAME}

Manual host script execution

We can go the Menu section Monitoring > Hosts, select the host, and click on it. In the menu, you will have an additional script available for the Docker hosts group Container action. This manual action is also available in some other frontend sections.

Fig 2. The available scripts for manual execution on the host

Once you click on the Container action, you will have several options available. You can start, restart, or stop the container.

Fig 3. Drop-down menu options for the script

You will have a confirmation window asking if this is the right action you want to perform.

Fig 5. Script output on successful execution.

The status can also be checked in the host’s latest data menu, once the metric is collected (1 minute for the master item). The item Container /zabbix-agent2: Running shows that this container is not running, and another item displays the exit code 0, which means the process stopped normally with no issue whatsoever.

Fig 6. The latest data for the Zabbix agent 2 container

Some items report the status in numerical format, e.g., 0 , 1 , 2, and so on. To make it human-readable, we use value maps, which display the value in a meaningful, human-friendly way. The screenshot below shows the value map for container health status. So, instead of looking at value 3 for container health (which is meaningless for us and will require reading the documentation) we will be shown value healthy (3).

Fig 7. Predefined value mapping on the template

Automating the container crash recovery

What if your container crashes for some reason? Well, you will get a problem event, which you can use to receive notifications about issues with containers. You can also automate the container recovery process. For example, create a trigger action that will restart the container 3 times with an interval of 2 minutes.

If it does not resolve the issue, only then send a message to the admin. There is no reason to repeatedly restart the service until the end of time – if a few attempts did not work, most likely it will require human intervention to solve the issue.

So here are the script parameters for the action operation:

● Script
▪ Name: Restart container
▪ Scope: Action operation
▪ Type: Script
▪ Execute on: Zabbix server
▪ Commands: curl -sS -X POST https://{$DOCKER.IP}:{$DOCKER.PORT}/containers{HOST.NAME}/restart --cert /etc/zabbix/ssl/certs/client-cert.pem --key /etc/zabbix/ssl/keys/client-key.pem --cacert /etc/zabbix/ssl/ca/ca.pem
▪ Description: Restart container
▪ Host group: Selected: Docker

Now we have to define a trigger action in order to make use of this script and send a notification to admin if that fails.

Let’s create a new trigger action:

● Action tab
▪ Name: Automatic container restart
▪ Type of calc: And (A and B)
▪ condition: Host group equals Docker
▪ condition: Event name contains Container has been stopped with error code
▪ Enabled: Check

● Operations tab
▪ Default operation step duration: 2m

Add operation
▪ Operation: Current host: Check
▪ Steps: 1 -3
Add operation
▪ Operation: Send message
▪ Steps: 4 – 4
▪ Custom message: Check
▪ Subject: Automated restart failed to bring container up: {HOST.NAME}
▪ Message: <b>Host: {HOST.NAME}<br>
           <b>Problem started at {EVENT.TIME} on {EVENT.DATE}<br>
           <b>Operational data: {EVENT.OPDATA}<br>
           <b>Original problem ID: {EVENT.ID}<br>

Fig 10. Action operation tab: new Operation step 1-3

Fig 11. Action operation tab: new step 4-4

The action should look like the screenshot below. Save it.

Testing container crash automatic recovery

I will stop the container with the command docker kill zabbix-agent2. The container has been stopped with an exit code different from 0, so when the item receives the data (in my case, after 1 minute) I get a problem event. The trigger action executes the script Container restart immediately, after 2 minutes, and after 4 minutes if the problem event has not been resolved.

Fig 13. Problem event about stopped container exit code 137

This script successfully restarted the container. The container is running again, and the problem event is resolved.

Fig 14. Resolved event with remote script execution

Let’s see if I am quick enough to kill the west proxy container repeatedly, before the item collects the data with the container running state. Well, I managed to be faster, so now you will see what happens when it “fails” to bring the container back to running state. Here in the Actions, we can see that Zabbix executed the script three times (I also stopped the container 3 times fast enough!) after which the action sent a notification to the admin about a failure to bring the container up with restarts.

Fig 15. Problem event about stopped container exit code 137

I have received the message that the container restart was unable to bring the container to a running state and requires human interaction to fix this.

Fig 16. Problem event notification in email

Summary

Now you know how to plan ahead and make use of the built-in capabilities of Zabbix to solve the issue without human intervention (where possible) and only get notifications when the automatic remediation attempt fails.

The post Zabbix and the Docker API, Part 3: Control appeared first on Zabbix Blog.

Zabbix and the Docker API, Part 2: Adapt

2026-04-29 Janis Eidaks

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/zabbix-and-the-docker-api-part-2-adapt/32912/

In this blog post, I will show you how to create a template for monitoring your Docker server with only API calls (without the Zabbix agent 2). Instead of creating a template, templated items, LLD rules, and trigger prototypes from scratch, we will adapt them from the existing template “Docker by Zabbix agent 2.”

How does the Zabbix agent 2 do it?

If you are wondering how the Zabbix agent 2 collects the data, you can look into the source code and see the magic behind the scenes: https://github.com/zabbix/zabbix/blob/master/src/go/plugins/docker/metrics.go.

In short, it uses a Unix non-TCP socket, makes the Docker API requests on the host, and returns JSON objects. Hey, we already know how to use it ourselves from the previous blog post, right?

For the Zabbix agent 2 to work with the Docker template, it needs access to the Unix socket, either by adding the user Zabbix to the group: Docker or running the Zabbix-agent2 as root.

Fig 1. The Zabbix agent 2 Docker plugin source code

How we will do it

I can adapt this template, improvise whenever I encounter a non-existing metric, and overcome any technical challenge with effort! For the most part, in the template we have a few Zabbix agent items that collect data in bulk and a lot of dependent items (and dependent item prototypes from the LLD rules).

The path forward is quite straightforward – we will clone the template and replace the Zabbix agent item type with the HTTP agent type item and add additional parameters shown below. I will also add additional user macros on the template, including the Docker server IP address, port, CA, SSL certificate, and key file names so that these can be adjusted on the host level.

Fig 2. The workflow of the template modifications

Cloning the template and changing the item type

First, clone the template “Docker by Zabbix agent 2” and give it a new name: “Docker stats by HTTP.” Next, modify the Templated Zabbix agent type item configuration with the following parameters:

Fig 3. The modification of the templated item configuration

Modify “Docker by HTTP” template items:

● Modify item: Get containers
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/containers/json?all=true     
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● Modify item: Get data_usage
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/system/df
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● Modify item: Get images
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/images/json
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

●Modify item: Get info
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/info
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● Modify item: Ping
  ▪ Type HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/_ping
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}
♯ Preprocessing (additional first step)
  ▪ Boolean to Decimals

We also need to modify the LLD rule configuration. Change item type from Zabbix agent type to HTTP agent type:

Fig 4. The modification of LLD discovery rule: containers discovery

Modify LLD rule: Containers discovery
● Discovery rule
  ▪ Type: HTTP agent
  ▪ Key: docker.containers.discovery[true]
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/containers/json?all=true     
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}
  ▪ Update interval: 1h
● LLD macros
  ▪ {#ID}   $.Id
  ▪ {#NAME} $.Names.first()

Modify LLD rule: Images discovery
● Discovery rule
  ▪ Type: Dependent item
  ▪ Master item: item> Get images
● LLD macros
  ▪ {#ID}   $.Id
  ▪ {#NAME} $.RepoTags

After that, we will also make changes in the LLD rule “Containers discovery” by modifying a few existing item prototypes (Zabbix agent type) and adding a new item. Below are the item prototypes that require modification:

● In LLD rule Containers discovery, modify parameters for item prototype: Container {#NAME}: Get info
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/containers{#NAME}/json       
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● In LLD rule Containers discovery, modify item prototype: Container {#NAME}: Get stats
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/containers{#NAME}/stats?stream=false  
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● In LLD rule Containers discovery, modify parameters for item prototype: Container {#NAME}: CPU percent usage
  ▪ Type: Calculated
  ▪ Formula: last(//docker.container_stats.cpu_usage.total.rate["{#NAME}"])/last(//docker.container_stats.system_cpu_usage.total.rate["{#NAME}"])*last(//docker.container_stats.online_cpus["{#NAME}"])*100
♯ Preprocessing (delete preprocessing step JSONPath)

● In LLD rule Containers discovery, add new item prototype: Container {#NAME}: System CPU total usage per second
  ▪ Name: Container {#NAME}: System CPU total usage per second
  ▪ Type: Dependent item
  ▪ Key: docker.container_stats.system_cpu_usage.total.rate["{#NAME}"]
  ▪ Type of information Numeric (float)
  ▪ Master item    prototype > Container {#NAME}: Get stats

♦ Tags (name:value)        
  ▪ component:cpu
  ▪ container:{#NAME}      

♯ Preprocessing

  ▪ JSONPath       $.cpu_stats.system_cpu_usage
  ▪ Change per second
  ▪ Custom multiplier: 1.0E-9

Cloning the template (again) and making minor modifications

Next, I will clone the template “Docker statistics by HTTP” and give the copy a new name – “Docker containers by HTTP.” In the template “Docker containers by HTTP,” delete the LLD rule Images discovery; delete templated items; from the LLD rule “Containers discovery” rule, delete all prototype entities (item prototypes), and add a filter in the LLD rule (shown below):

In LLD rule Containers discovery rule, in Filter tab: add additional filter option
Filters [type of calculation: A and B and C]
   ▪ {#ID} matches {HOST.HOST}

I have also created a host group “Docker” where the discovered container hosts will be added. In the template “Docker statistics by HTTP” delete all item prototypes in the LLD rule “Containers discovery.” We will create a Host prototype in the LLD discovery rule “Containers discovery” – the parameters are shown below:

Host prototype in LLD rule: Containers discovery
  ▪ Host name:     {#ID}
  ▪ Visible name:  {#NAME}
  ▪ Templates:     Docker containers by HTTP

Fig 5. Host prototype settings in LLD rule: Containers discovery

Fig 6. The cloned and modified templates

Creating a host and linking the template

Now all that is left is to create a host and link a template: “Docker statistics by HTTP.” Do not forget to add the correct Docker IP address or DNS name in the user macro.

I have created a new host “Docker server,” linked a template, and modified the user macro for the Docker IP address. This host will collect Docker overall statistics. After the LLD discovery execution, the container names will be automatically discovered and container hosts will be created with a linked template.

Fig 7. The Discovered container hosts with linked templates

If for some reason you are monitoring multiple Docker instances, you could have the same container names discovered, which will lead to LLD errors, as there can’t be hosts with the same name (container ID) or visible name (container name). Quick solution – for each Docker instance, add a prefix to each container name. Another solution – don’t split the template into two parts, then the items will be discovered under the same host, and you will not have this issue.

The Docker server host shows general information about the Docker server’s overall state and status:

Fig 8. Docker server hosts the latest data

A host will be created automatically for each discovered container and will collect the container-specific performance metrics:

Fig 9. The container: zabbix-agent2 latest data

Summary

Now you and I know a little bit more about how Zabbix agent2 is collecting Docker metrics. This blog post has shown you how to improvise and adapt existing templates with different data collection methods. Zabbix is a very versatile tool that you can use in multiple ways to get the data if you have some technical constraints. The included template can also be used as is, or you can modify it to suit your needs.

The post Zabbix and the Docker API, Part 2: Adapt appeared first on Zabbix Blog.

Zabbix and the Docker API, Part 1: Inspect

2026-04-22 Janis Eidaks

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/zabbix-and-the-docker-api-part-1-inspect/32860/

In this blog post, I will show you how to configure Zabbix to securely gather Docker API metrics using the Zabbix HTTP agent item with certificate authentication. This guide will cover configuring the Docker API and the Zabbix server side to gather data more securely.

Getting the data to Zabbix from the Docker API

By default, Docker API uses a non-network socket for security reasons, and there are several valid reasons for this. It is not advised to expose your Docker environment over TCP to localhost, and even less to the internet. Exposing the Docker API without any security to the internet is just inviting hackers for free lunch, as anyone (bots included) who can access your Docker API will also be able to do malicious operations with it (make changes, launch malicious containers, try to take over your environment, and do a lot of harm in general) !

So, make sure to harden your environment’s security and use this guide at your own judgment. Also, set up your firewall so only the Zabbix server can access the Docker API port! By default, you can check if the Docker service is active and if you can get a response to the Docker API by running the curl command:

# systemctl is-active docker
# curl --silent --show-error --unix-socket /var/run/docker.sock http://localhost/info |jq

Fig 1. Output of the Docker API call in CLI

Generating the certificates for the Docker and Zabbix server

You can use the right tool for the job, such as an HTTP agent for the Docker API requests with proper certificate authentication. You will require the CA private key and CA certificate; private key and certificate for the Docker server; private key and certificate for the Zabbix server (for simplicity, you can generate all of them on the Docker server and copy the appropriate files to the Docker server and Zabbix server directories).

A guide you can follow to generate the certificate files is located here: https://docs.docker.com/engine/security/protect-access/#use-tls-https-to-protect-the-docker-daemon-socket.

Deploying the certificate files and configuring the services

On the Docker server, copy the CA and server certificate files to /etc/docker directory:

# cp -v {ca,server-cert,server-key}.pem /etc/docker

The Docker daemon also requires JSON configuration with additional settings (allow TCP/ Unix socket, TLS options):

# nano /etc/docker/daemon.json

{
  "hosts": ["tcp://0.0.0.0:2376","unix:///var/run/docker.sock"],
  "tls": true,
  "tlsverify": true,
  "tlscacert": "/etc/docker/ca.pem",
  "tlscert": "/etc/docker/server-cert.pem",
  "tlskey": "/etc/docker/server-key.pem"
}

We will have to add the Docker service override to remove the Unix socket from the Docker systemd service, then reload the daemon, and restart the Docker service.

# mkdir -p /etc/systemd/system/docker.service.d

# nano /etc/systemd/system/docker.service.d/override.conf

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd

# systemctl daemon-reload
# systemctl restart docker

On the Zabbix server side, create directories for certificate files. Then, copy the relevant certificate files from the Docker server. In my case, I generated all of the certificate files on the Docker host (replace docker in the scp command with IP/DNS name of the Docker server): ca.pem, client-cert.pem, client-key.pem, to their respective directories and change their permissions.

# mkdir -pv /etc/zabbix/ssl/{ca,certs,keys}
# scp root@docker:/root/dockercerts/ca.pem /etc/zabbix/ssl/ca/
# scp root@docker:/root/dockercerts/client-cert.pem /etc/zabbix/ssl/certs/
# scp root@docker:/root/dockercerts/client-key.pem /etc/zabbix/ssl/keys/
# chmod -v 0400 /etc/zabbix/ssl/keys/client-key.pem
# chmod -v 0444 /etc/zabbix/ssl/ca/ca.pem /etc/zabbix/ssl/certs/client-cert.pem
# chown zabbix:zabbix -R /etc/zabbix/ssl

Check if you can get data in the Zabbix server from the Docker server with HTTPS request (replace $HOST with your Docker server address):

# curl -sS https://$HOST:2376/info \
  --cert /etc/zabbix/ssl/certs/client-cert.pem \
  --key /etc/zabbix/ssl/keys/client-key.pem \
  --cacert /etc/zabbix/ssl/ca/ca.pem |jq

Fig 2. Executing the HTTPS request to the Docker server from the Zabbix server machine

If everything works so far, then it is time to modify the Zabbix server configuration file and specify the location of the certificate file directories. After that, restart the Zabbix server service.

# nano /etc/zabbix/zabbix_server.conf

SSLCertLocation=/etc/zabbix/ssl/certs
SSLKeyLocation=/etc/zabbix/ssl/keys
SSLCALocation=/etc/zabbix/ssl/ca

# systemctl restart zabbix-server

You will also need to copy the Docker CA file to the trusted CA directory and update the CA list.

# cd /
# cp /etc/zabbix/ssl/ca/ca.pem /etc/pki/ca-trust/source/anchors/
# update-ca-trust extract

Fig 3. The location of certificate files in the directories for each server

Configuring the monitoring in the Zabbix frontend

If you have read this far and decided that this is too much work or this approach is not feasible in your environment (company policy or some other technical limitation), don’t be discouraged so fast! There is another way to get the metrics without changing the Docker configuration, creating certificates, and configuring the Zabbix server config file – simply use an SSH agent-type item to gather the data.

To prepare for both approaches, I will create a host with multiple user macros, which will store the IP address, port, SSH user, SSH password, and SSL certificate information.

The easy way: SSH agent items

However, what to do if the company policy prohibits installing additional applications to gather data, such as the Zabbix agent (or changing Docker configuration settings, as in this case)? In this instance, you can use other, seemingly simpler ways to gather metrics, such as using the SSH agent item.

If the only tool you have is a hammer (SSH access), you tend to see every problem as a nail. The old adage “do not fix what is not broken” is still prevalent in this era! In that case, create an SSH agent-type item. Specify the IP address and SSH port in the item key, the username and password for the Docker host, and specify a command to gather the data. For those fields, I will use the previously defined user macros.

Here is an example of the SSH item configuration:

Host: Docker server items 
Item #1
  ▪ Name:          Get info ssh
  ▪ Type           SSH agent
  ▪ Key:           ssh.run[docker.infos,{$DOCKER.IP}]  
  ▪ Type of inf:   text
  ▪ Username:      {$SSH.USER}
  ▪ Password:      {$SSH.PASSWORD}
  ▪ Ex. script:    curl --unix-socket /var/run/docker.sock http://localhost/info

Fig 6. Example of SSH agent item configuration for executing a script on the Docker server

You can also test the item and obtain the same data in JSON format, shown in Fig. 1.

The right way: HTTP agent

For the other approach, we will be using an HTTP agent item to collect the data in bulk, using Docker API calls. For this, I don’t need to install the Zabbix agent on the Docker server. The authentication of this item will be performed using the certificates that have been copied over. Here are the important parameters in the item:

Host: Docker server items 
Item #1
  ▪ Name:             Get info
  ▪ Type              SSH agent
  ▪ Key:              docker.info    
  ▪ Type of inf:      text
  ▪ URL:              https://{$DOCKER.IP}:{$DOCKER.PORT}/info
  ▪ SSL verify peer: check
  ▪ SSL verify host: check
  ▪ SSL cert. file:  {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file:    {$SSL.KEY.FILE}

Do not forget to test the item (collected data should be the same as in Fig. 2) and add the item. If you have also encrypted the client private key (client-key.pem), you will also need to provide an SSL key password in the item configuration.

Fig 8. Example of the configured HTTP agent item

Fig 9. HTTP agent item collecting the data

Extracting the data

Now we can extract the important metrics by creating dependent items using the master item: Get info. Add a few dependent items to extract metrics, such as the total count, running, stopped, and paused containers. Item configuration parameters are given below the dependent item examples. The item “Containers running” parameter screenshots are shown below, together with the configuration parameters listed.

Fig 10. Dependent item tab to get the number of running containers

Tagging an item will also make your life easier for filtering when you have a legion of items.

Fig 11. Dependent item tag tab to get the number of running containers

In the preprocessing tab, we can use the JSONPath preprocessing step to extract the number of running containers from the master item.

Fig 12. Dependent item preprocessing tab to get the number of running containers

Docker Host items
● Item #1
  ▪ Name: 	Containers running	
  ▪ Type 		Dependent item
  ▪ Key: 		docker.containers.running	
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item	Docker: Get info
  ▪ Units: 	!containers
♦ Tags (name:value) 	
  ▪ component:containers	
♯ Preprocessing
  ▪ JSONPath  	$.ContainersRunning

● Item #2
  ▪ Name: 	Containers paused	
  ▪ Type 		Dependent item
  ▪ Key: 		docker.containers.paused	
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item	Docker: Get info
  ▪ Units: 	!containers
♦ Tags (name:value) 	
  ▪ component:containers	
♯ Preprocessing
  ▪ JSONPath  	$.ContainersPaused

● Item #3
  ▪ Name: 	Containers stopped	
  ▪ Type 		Dependent item
  ▪ Key: 		docker.containers.stopped	
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item	Docker: Get info
  ▪ Units: 	!containers
♦ Tags (name:value) 	
  ▪ component:containers	
♯ Preprocessing
  ▪ JSONPath  	$.ContainersStopped

● Item #4
  ▪ Name: 	Containers total	
  ▪ Type 		Dependent item
  ▪ Key: 		docker.containers.total	
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item	Docker: Get info
  ▪ Units: 	!containers
♦ Tags (name:value) 	
  ▪ component:containers	
♯ Preprocessing
  ▪ JSONPath  	$.Containers

Creating the trigger

I can also configure a trigger to receive a problem event in case some containers are not running. The screenshot of the trigger and parameter configuration is shown below.

Trigger
◘ Trigger 
  ▪ Name: 		Some containers are not running
  ▪ Operational data: 	Total: {ITEM.LASTVALUE1}, Running: {ITEM.LASTVALUE2}
  ▪ Severity: 		Warning
  ▪ Expression: 		last(/Docker server/docker.containers.total)last(/Docker server/docker.containers.running)
  ▪ PROBLEM event generation mode: Single
  ▪ OK event closes: All problems

Getting more data

Docker Engine also includes previous API versions. If no version of the API is specified in the URL, then the latest installed version will be used (using the API without a version is deprecated and will be removed in a future release). So even if you have the latest Docker installed (and you should always update to the latest version!), you can still use the older API calls by specifying the version (but once again, check what works).

Docker API offers several API calls that can be used to collect information about containers, images, container performance statistics, networks, volumes, or make changes to them.

Also, for more API calls, please explore this page: https://docs.docker.com/reference/api/engine/latest/.
As an example, I will create another item to gather specific container information. The item configuration will differ from the one in the example in Fig.8 with the following parameters: different URL, item name, and key.

Here is an example of the ULR field (replace {$CONTAINER} with the existing container name):

https://{$DOCKER.IP}:{$DOCKER.PORT}/containers/{$CONTAINER}/json

Fig 14. HTTP agent item to get low-level information about a specific container: tomcat

You can also get the container performance data with a different URL. The item configuration will differ from the one in an example in Fig.8 with the following parameters: URL, item name and key. Here is an example of ULR field (replace {$CONTAINER} with the existing container name):

https://{$DOCKER.IP}:{$DOCKER.PORT}/containers/{$CONTAINER}/stats?stream=false

Fig 15. HTTP agent item to get performance information about a specific container: zabbix-server-mysql

Testing the trigger

We can test if the data returned by the Docker API is as it seems, right? I have five containers created using the ‘docker run’ command, and one using the ‘docker compose’ command. Let’s stop the container made from the ‘docker run’ command and check if it will be reflected in the collected metrics.

Fig 16. The latest item data when stopping a Docker Compose container

As you can see in Figure 13, the stopped container shows up in the metrics collected by Zabbix through Docker API and in the Docker CLI. The Docker host item shows 1 stopped container and 5 running containers; the total number of containers is 6.

If you use the command “docker compose down” instead, the container will be stopped and removed altogether. That means, the total number of containers will also decrease by one, along with its status, as shown in Fig. 17. Therefore, make sure you understand what each command does and how it will impact your monitoring data.

Fig 17. The latest item data when using Docker Compose down for a container

In summary

Now you know more about how to collect the data from Docker using HTTP requests. Similar approaches can also be used to collect data from other applications through an API. You can select what metrics you want to extract, create triggers, graphs, or make a template if you wish.

The post Zabbix and the Docker API, Part 1: Inspect appeared first on Zabbix Blog.

Decoding Zabbix Proxy Traffic for Faster Troubleshooting

2026-01-27 Kaspars Mednis

Post Syndicated from Kaspars Mednis original https://blog.zabbix.com/decoding-zabbix-proxy-traffic-for-faster-troubleshooting/31898/

Usually, it is enough to simply look at the Zabbix proxy administration page or proxy health metrics to perform basic proxy troubleshooting. However, there are situations when a deeper look is required.

Today, we will examine the Zabbix server proxy communication and learn how to interpret the internal communication protocol.

Understanding the protocol

Zabbix communication protocol

Zabbix components use TCP for communication, and information is encoded in JSON. How do you distinguish Zabbix communication packets? There are a few main filters you need to apply:

Protocol: TCP
Port: 10051 or 10050 (depending on whether components are active or passive)
Packet: Starts with ZBXD or 5A 42 58 44 in HEX

On older versions, it was simple to capture and read Zabbix packets in plain text. Starting with Zabbix 4.0.0, mandatory traffic compression was implemented. This greatly reduces network traffic – roughly by 10× with negligible CPU overhead, but it also makes the traffic unreadable to humans.

A modern Zabbix communication packet looks like this:

5a425844038200000097000000789c2dcccb0e83201085e15731b33606b90a8fe20ec631256da4056a6c9abe7be965fb7f27e709996e772a151c5c733a1edde2ab871e4ee9db661f423cba1f79ac71a786854a89696bbe7223e5b2326b75e01c199368510b42cf68f2eaf3b453fe8fcdc0063eb68497846770a3d1c25a698cea612be0b43642a9c9b2d71b6c5d2cfd

Not very human-friendly, right? In the following sections we will capture and decompress this communication packet step by step.

Capturing traffic

There are multiple tools available for this purpose, but we will use Wireshark – one of the most popular and widely used packet analysis tools. It provides a nice graphical interface for Windows and Linux, but we will use the command-line version, since most troubleshooting is performed over an SSH session. The system used in this example is CentOS Stream 9, but the commands should work on other Linux distributions with only minor syntax adjustments.

First, install the tool:

dnf install wireshark-cli

This installs the tshark command-line utility. After that, change your working directory to a location where you can write files. In this example, we will use /tmp:

cd /tmp

Next, let’s capture some traffic between the Zabbix server and an active proxy:

tshark -i eth0 -f "host <ZABBIX SERVER IP> and host <ZABBIX PROXY IP> \
and tcp port 10051" -w zabbix_stream.pcap

Explanation of parameters:

-i eth0 – listen on interface eth0 (specify a different interface if needed)
<ZABBIX SERVER IP> – replace with the Zabbix server IP address
<ZABBIX PROXY IP> – replace with the Zabbix proxy IP address
tcp port 10051 – capture TCP packets on port 10051 (Zabbix trapper)
-w zabbix_stream.pcap – write captured output to a file

Let this run for a couple of minutes to collect some raw traffic data. Press CTRL + C to stop the capture.

Analyzing capture file

Now we have captured a *.pcap file that contains multiple TCP streams. A TCP stream represents a single TCP connection. Since Zabbix proxies do not keep persistent connections and instead open a new connection whenever needed, a Zabbix active proxy typically produces the following streams:

Data sender – sends collected values every second (by default)
Configuration syncer – downloads configuration updates every 10 seconds (by default)

To view the contents of the *.pcap file, run:

tshark -r zabbix_stream.pcap -q -z conv,tcp

Example output:

TCP Conversations
Filter:<No Filter>
                                   |      <-    ||      ->    ||     Total   |Relative|
                                   |Frames Bytes||Frames Bytes||Frames Bytes |Start   |       
10.10.0.2:57850 <-> 10.20.0.5:10051 5 2,512bytes  6 547bytes    11 3,059bytes 0.0000   
10.10.0.2:57860 <-> 10.20.0.5:10051 5 399bytes    5 516bytes    10 915bytes   0.4700  
10.10.0.2:57864 <-> 10.20.0.5:10051 5 399bytes    5 521bytes    10 920bytes   1.4768  
10.10.0.2:57876 <-> 10.20.0.5:10051 5 399bytes    5 570bytes    10 969bytes   2.4829   
10.10.0.2:57878 <-> 10.20.0.5:10051 5 399bytes    5 522bytes    10 921bytes   3.4882   
10.10.0.2:46628 <-> 10.20.0.5:10051 5 399bytes    5 527bytes    10 926bytes   4.4935   
10.10.0.2:46642 <-> 10.20.0.5:10051 4 333bytes    6 590bytes    10 923bytes   5.4992   
10.10.0.2:46648 <-> 10.20.0.5:10051 5 399bytes    5 478bytes    10 877bytes   6.5047   
10.10.0.2:46662 <-> 10.20.0.5:10051 5 399bytes    5 480bytes    10 879bytes   7.5097

We can print packets in chronological order, including stream numbers:

tshark -r zabbix_stream.pcap -T fields \
-e tcp.stream -e frame.number -e frame.time_relative -e frame.len

Column meaning in example output:

Stream number
Frame number
Relative timestamp from the start of capture
Frame size in bytes

0 1  0.000000000 76
0 2  0.000005109 76
0 3  0.000078403 68
0 4  0.000079579 68
0 5  0.000280946 209
0 6  0.000283835 209
0 7  0.001188322 68
0 8  0.001189912 68
0 9  0.001421210 68
0 10 0.001422856 68
1 11 1.003582601 76
1 12 1.003588266 76
1 13 1.003646494 68
1 14 1.003647585 68
1 15 1.003741654 256
1 16 1.003758183 256
1 17 1.004531106 68
1 18 1.004532827 68
1 19 1.004973531 68
.....

To include the payload (Zabbix communication), add the -e tcp.payload field:

tshark -r zabbix_stream.pcap -T fields \
-e tcp.stream -e frame.number -e frame.time_relative -e frame.len -e tcp.payload

Example (truncated for readability):

0 1  0.000000000 76
0 2  0.000005109 76
0 3  0.000078403 68
0 4  0.000079579 68
0 5  0.000280946 209 5a425844038000000096000000789c2dca4d0e82301040e1ab90591352fb3703477137d3d6483454692518e3dd6dd4edfbde0bd6747fa4526182db9af76717b932f470cedf76649179ef7ec4a1ce5b6a585229735e9a2bf12aa2481c89299032c2ceb21754a44fda609bb7b4fe671cece05a09d71c2e301dd05b4548daf4b014984667b52734f6fd013eac2c96
0 6  0.000283835 209 5a425844038000000096000000789c2dca4d0e82301040e1ab90591352fb3703477137d3d6483454692518e3dd6dd4edfbde0bd6747fa4526182db9af76717b932f470cedf76649179ef7ec4a1ce5b6a585229735e9a2bf12aa2481c89299032c2ceb21754a44fda609bb7b4fe671cece05a09d71c2e301dd05b4548daf4b014984667b52734f6fd013eac2c96
0 7  0.001188322 68
0 8  0.001189912 68
0 9  0.001421210 68
0 10 0.001422856 68
......

Not all frames contain payload — the empty ones represent TCP handshakes and other control packets. We are interested only in frames containing payload, because this is where Zabbix data lives.

Analyzing payload

If you take a closer look, each payload starts with a sequence of 5a 42 58 44 – or “ZBXD” in ASCII. This is the Zabbix packet signature and confirms that we have captured the correct traffic.

Example:

5a 42 58 44 – Zabbix packet signature ZBXD
03 – Flags (0x01 Zabbix protocol + 0x02 compression)
af 00 00 00 – Data length
f0 00 00 00 – Length of uncompressed data

The next header is: 78 9c which indicates zlib compression. After this comes the compressed JSON data we are interested in. More information can be found within Zabbix documentation here.

Let’s extract only the payload with command:

tshark -r zabbix_stream.pcap -T fields -e tcp.payload -E occurrence=f \
| grep -v '^$'

-T fields: output only selected fields
-e tcp.payload: get the payload of each TCP frame
-E occurrence=f: include all occurrences per frame
grep -v ‘^$’: remove empty lines (frames with no payload)

Output example:

5a425844038000000096000000789c2dca4d0e82301040e1ab90591352fb3703477137d3d6483454692518e3dd6dd4edfbde0bd6747fa4526182db9af76717b932f470cedf76649179ef7ec4a1ce5b6a585229735e9a2bf12aa2481c89299032c2ceb21754a44fda609bb7b4fe671cece05a09d71c2e301dd05b4548daf4b014984667b52734f6fd013eac2c96                                                                                5a425844038000000096000000789c2dca4d0e82301040e1ab90591352fb3703477137d3d6483454692518e3dd6dd4edfbde0bd6747fa4526182db9af76717b932f470cedf76649179ef7ec4a1ce5b6a585229735e9a2bf12aa2481c89299032c2ceb21754a44fda609bb7b4fe671cece05a09d71c2e301dd05b4548daf4b014984667b52734f6fd013eac2c96                                                                                5a42584403af000000f0000000789c658ecb0e823014447f85dc3521853e6edb4fd1b868a1c646b44a0bc110fedd22ec5cce9ce4cc2c30b8f7e862020daf21cc9fa233c94009b7f0eb4ec65a3f173b326df293cb30ba187d78664eac201d5adb2969642b09b58633232c12d95c1b8a9bc9c7148643a

Decompressing payload

First, let’s save the payload to a file:

tshark -r zabbix_stream.pcap -T fields -e tcp.payload -E occurrence=f \
| grep -v '^$'  > zabbix_payload.hex

Next, create a python script named decompress.py.

#!/usr/bin/python3
import zlib

hex_file = "zabbix_payload.hex"
ZBXD_HEADER_LEN = 26 # 13 bytes * 2 hex chars per byte

with open(hex_file, "r") as f:
  for line_number, line in enumerate(f, 1):
    line = line.strip()
    if not line:
      continue

    # Remove Zabbix header
    if line.startswith("5a425844"):
      payload_hex = line[ZBXD_HEADER_LEN:]
    else:
      payload_hex = line

    # Convert hex to bytes
    try:
      payload_bytes = bytes.fromhex(payload_hex)
    except ValueError as e:
      print(f"Line {line_number}: Invalid hex, skipping ({e})")
      continue

    # Decompress using zlib
    try:
      decompressed = zlib.decompress(payload_bytes)
    except zlib.error as e:
      print(f"Line {line_number}: Decompression error ({e})")
      continue
  
    print(f"Line {line_number}: {decompressed}")

Make the file executable:

chmod +x decompress.py

Execute the file:

./decompress.py

The script will output decompressed Zabbix traffic:

Line 59: b'{"request":"proxy data","host":"Zabbix proxy active","session":"fbdb545d8250bb4c9b2341cc8ca055f1","history data":[{"id":13,"itemid":50454,"clock":1764172374,"ns":946257883,"value":"[{\\"{#IFNAME}\\":\\"lo\\"},{\\"{#IFNAME}\\":\\"eth0\\"}]"}],"version":"7.4.5","clock":1764172375,"ns":432069960}'
Line 60: b'{"upload":"enabled","response":"success","tasks":[{"type":6,"clock":1764172373,"ttl":3600,"itemid":50454}]}'
Line 61: b'{"request":"proxy data","host":"Zabbix proxy active","session":"fbdb545d8250bb4c9b2341cc8ca055f1","version":"7.4.5","clock":1764172375,"ns":438122213}'
Line 62: b'{"upload":"enabled","response":"success"}'
Line 63: b'{"request":"proxy config","host":"Zabbix proxy active","version":"7.4.5","session":"fbdb545d8250bb4c9b2341cc8ca055f1", "config_revision":18611,"proxy_secrets_provider":0}'
Line 64: b'{"data":{},"config_revision":18613}'

Here every line represents a request from a Zabbix active proxy or Zabbix server response. It is easy to distinguish two communication types:

Request proxy data – Proxy sends collected values
Request proxy config – Proxy checks its configuration revision and downloads configuration changes if required

Recap

It is required to run only three commands in this setup to read uncompressed communications:

tshark -i eth0 -f "host <ZABBIX SERVER IP> and host <ZABBIX PROXY IP> \
and tcp port 10051" -w zabbix_stream.pcap

tshark -r zabbix_stream.pcap -T fields -e tcp.payload -E occurrence=f \
| grep -v '^$' > zabbix_payload.hex

./decompress.py

A more human-readable format

Can we improve it? Absolutely! Let’s pair requests with their corresponding responses for easier parsing, and then output the data as formatted JSON. First, capture the data:

tshark -i eth0 -f "host <ZABBIX SERVER IP> and host <ZABBIX PROXY IP> \
and tcp port 10051" -w zabbix_stream.pcap

Next, extract the data into a CSV while keeping the stream number:

tshark -r zabbix_stream.pcap -T fields -e tcp.stream -e tcp.payload \
-E occurrence=f -E separator=, -E quote=d, -Y 'tcp.payload && tcp.payload != ""' \
> zabbix_payload.csv

Now, the CSV contains both the stream number and the payload for each packet.

"2","5a42584403aa000000dd000000789c458d410e83201444af62fe9a1814a896a3b4e9e283df9494480bd4688c772f694dba9d37336f8348af37a50c1a9e312c6b35604660700fdfec82c6b8a5fa21b4d9cd5460a2945c980a1fcd60945443df2a6e8cb467d30ad958db5be44a8d4d29bb29531cd15285333a8fc6799757d0d7ed8fdc005a080647c31368ce80620cb14860bf3198291eceae96b52ac7d607fb00dd7427d974ad506531a5f2c3c599ab9ecbfd038a0944ee" "2","5a425844033000000029000000789cab562a2dc8c94f4c51b2524acd4b4cca494d51d2512a4a2d2ec8cf2b4e050a16972627a716172bd502002b010e61" "3","5a42584403db0000003d010000789c658fdd6ac3300c855f25e8da143bb6f2e317196cecc23f0a33f3e2cd76434be9bbcf4d03bbd89584bea3a3a31b64fa3953a9a0e13ba7cbb5f3a61a60f091f6d9abb1365cba2732ae868d1a2c544a486be38bf51615faa9476ead72b3eda512ce4dce70c4453471582be5c538eacc66423436c450afa0df6e7f2878d05232381491400b069473caed08dcdf5ba0506aca47be7dd9efa250e9ebd12257c819b898dc6703e3a0c4d8cbc7682da06735e1340e02196c269e9b3fbc90ed0ae58df2eedfeaf1d378522784ff56e2692545afe6990fc3fd17684060c8" "3","5a425844033000000029000000789cab562a2dc8c94f4c51b2524acd4b4cca494d51d2512a4a2d2ec8cf2b4e050a16972627a716172bd502002b010e61"

Next, let’s create a slightly modified Python script to display the entries per stream. Name it streams.py:

#!/usr/bin/python3

import csv
import zlib
import json

csv_file = "zabbix_payload.csv"
ZBXD_HEADER_LEN = 26 # 13 bytes * 2 hex chars per byte
streams = {}
with open(csv_file, "r") as f:
  reader = csv.reader(f)
  for row_number, row in enumerate(reader, 1):
    if len(row) < 2:
       continue

    stream_id = row[0].strip().strip('"')
    hexdata = row[1].strip().strip('"')

    if not hexdata:
      continue

    # Remove Zabbix header
    if hexdata.startswith("5a425844"):
      hex_payload = hexdata[ZBXD_HEADER_LEN:]
    else:
      hex_payload = hexdata

    # Convert hex to bytes
    try:
      payload_bytes = bytes.fromhex(hex_payload)
    except ValueError as e:
      print(f"[Line {row_number}] Invalid hex: {e}")
      continue

    # Decompress
    try:
      decompressed = zlib.decompress(payload_bytes)
    except zlib.error as e:
      print(f"[Line {row_number}] Decompression error: {e}")
      continue

    # Store in the stream bucket
    streams.setdefault(stream_id, []).append(decompressed)

# ---- OUTPUT SECTION ----

print("\n===== STREAM PAIRS =====\n")

for stream_id, messages in streams.items():
  print(f"=== Stream {stream_id} ===")
  for i, msg in enumerate(messages):
    label = (
      "Request:" if i == 0
      else "Response:" if i == 1
      else f"Extra message #{i+1}:"
    )
    print(label)
    text = msg.decode("utf-8")

    # Try to pretty-print JSON
    try:
      parsed = json.loads(text)
      pretty_json = json.dumps(parsed, indent=4, ensure_ascii=False)
      print(pretty_json)
    except json.JSONDecodeError:
    # fallback: print raw text
      print(text)
    print()

Make the file executable:

chmod +x streams.py

Execute the file:

./streams.py

The script will output decompressed Zabbix traffic in a parsed JSON format:

=== Stream 0 ===
Request:
{
  "request": "proxy data",
  "host": "Zabbix proxy active",
  "session": "fbdb545d8250bb4c9b2341cc8ca055f1",
  "interface availability": [
    {
      "interfaceid": 33,
      "available": 0,
      "error": ""
    }
  ],
  "version": "7.4.5",
  "clock": 1764172350,
  "ns": 303905804
}
Response:
{
  "upload": "enabled",
  "response": "success"
}

=== Stream 1 ===
Request:
.......

You’ll notice that typical communication produces two entries per stream – one request from the Zabbix proxy and one response from the Zabbix server. With this approach, it’s much easier to understand and troubleshoot the communication – all traffic is now grouped into request-response pairs and presented in a clean, formatted way.

Live data

And finally — can we make all of this run live? Absolutely, with a little help from our third Python script. The previous two examples walked through the workflow step by step: capture → extract payload → decompress. Now everything comes together in a single script that handles the entire process for you.

Create a new file named live.py:

#!/usr/bin/python3

import subprocess
import zlib
import json
from datetime import datetime

ZBXD_HEADER_LEN = 26 # 13 bytes * 2 hex chars

# === Configurable parameters ===
SRC_IP = "161.35.217.186"
DST_IP = "134.209.233.72"
TCP_PORT = "10051"
INTERFACE = "eth0"

tshark_cmd = [
  "tshark",
  "-i", INTERFACE,
  "-l",
  "-f", f"host {SRC_IP} and host {DST_IP} and tcp port {TCP_PORT}",
  "-T", "fields",
  "-e", "tcp.stream",
  "-e", "tcp.payload",
  "-E", "separator=,",
  "-E", "quote=d",
  "-E", "occurrence=f",
  "-Y", "tcp.payload && tcp.payload != \"\""
]

proc = subprocess.Popen(
  tshark_cmd,
  stdout=subprocess.PIPE,
  stderr=subprocess.DEVNULL,
  text=True
)

seen_streams = set() # track streams we've already printed

for line in proc.stdout:
  line = line.strip()
  if not line:
    continue

  # Split CSV (stream_number, payload_hex)
  try:
    stream_num, payload_hex = line.split(",", 1)
    payload_hex = payload_hex.strip('"')
  except ValueError:
    continue

  # Only print timestamp once per stream
  if stream_num not in seen_streams:
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]
    print(f"\n=== [{timestamp}] Stream {stream_num} ===")
    seen_streams.add(stream_num)

  # Remove Zabbix header
  if payload_hex.startswith("5a425844"):
    payload_hex = payload_hex[ZBXD_HEADER_LEN:]

  # Convert hex to bytes
  try:
    payload_bytes = bytes.fromhex(payload_hex)
  except ValueError:
    continue

  # Decompress
  try:
    decompressed = zlib.decompress(payload_bytes)
  except zlib.error:
    continue

  # Pretty print JSON if possible
  try:
    json_obj = json.loads(decompressed)
    pretty = json.dumps(json_obj, indent=2)
    print(pretty)
  except json.JSONDecodeError:
    print(decompressed)

Make the file executable:

chmod +x live.py

Execute the file:

./live.py

And that’s it – your script now watches live proxy traffic and streams the output as JSON. Pretty cool, right?

=== [2025-11-27 16:59:31.593] Stream "0" ===
{
  "request": "proxy data",
  "host": "Zabbix proxy active",
  "session": "fbdb545d8250bb4c9b2341cc8ca055f1",
  "history data": [
    {
      "id": 73726,
      "itemid": 50459,
      "clock": 1764262769,
      "ns": 947018320,
      "value": "0"
    },
    {
      "id": 73727,
      "itemid": 50450,
      "clock": 1764262770,
      "ns": 947145177
    }
  ],
  "version": "7.4.5",
  "clock": 1764262770,
  "ns": 961735298
}
{
  "upload": "enabled",
"  response": "success"
}
.....

Final notes

The example scripts provided here are for demonstration purposes only, tested in a small demo environment. While the same principles apply to larger setups, keep in mind that proxies in production can handle hundreds or even thousands of new values per second (NVPS), which significantly increases the payload volume. Also, all examples assume a Zabbix proxy running in active mode – passive proxies communicate slightly differently. A similar approach can be used to monitor Zabbix Agent communications.

So, what valuable information can you actually gather from Zabbix proxy ⇄ Zabbix Server communication?

The types of data sent from proxy to server
Configuration updates and their contents
Test and Execute Now tasks
Discovery and Autoregistration data

If you’re interested in exploring discovery, autoregistration, encryption, or other aspects of Zabbix’s internal communication, feel free to leave a comment!

The post Decoding Zabbix Proxy Traffic for Faster Troubleshooting appeared first on Zabbix Blog.

24/7 Alerting and Two-Way Integration with Zabbix and SIGNL4

2025-12-09 Ronald Czachara

Post Syndicated from Ronald Czachara original https://blog.zabbix.com/24-7-alerting-and-two-way-integration-with-zabbix-and-signl4/31866/

It’s a familiar story for many IT operations teams: a critical server went down overnight, but the alert was buried in someone’s inbox. By the time anyone noticed, valuable time was lost, SLAs were breached, and the team spent the next morning explaining why an email hadn’t been seen. Email (or even SMS text) alone simply wasn’t reliable enough for something as urgent as incident alerts.

The turning point came when the team decided to integrate SIGNL4 with Zabbix. Setup was fast – within minutes, alerts that once hid in crowded inboxes were now reaching the right on-call engineer – loud, clear, and actionable. Instead of reacting late, the team was responding in real time and the night shifts suddenly felt a lot less stressful.

Integration overview and two-way communication

The SIGNL4 integration leverages a Zabbix media type to seamlessly send event data from Zabbix to SIGNL4. Once configured, Zabbix alerts are instantly transformed into mobile push notifications, ensuring rapid delivery and clear visibility for on-call teams.

Beyond alerting, the integration also supports bidirectional status updates between the two systems – including acknowledgements, closures, and annotations. When an on-call engineer acknowledges an alert in the SIGNL4 mobile app, the status is automatically reflected in the Zabbix dashboard.

Likewise, when Zabbix detects recovery (status UP), it triggers an automatic update to close the corresponding alert in SIGNL4. This real-time synchronization keeps both platforms perfectly aligned, maintaining consistent alert and recovery states without any manual effort.

Configuration steps

In the Zabbix web portal go to “Alerts” -> “Media types.”

Find the SIGNL4 media type, enable it, and enter your SIGNL4 team or integration secret in the parameter “teamsecret.” Alternatively, you can leave the default ({ALERT.SENDTO}) and enter the SIGNL4 team secret into the “Send to” parameter of your user.

Update the settings:

In the media type list click the button “Test” for the SIGNL4 media type to send a test alert. You will receive an alert in your SIGNL4 mobile app.

Under “User settings” -> “Profile” go to “Media” and add the SIGNL4 media type here. Adapt the alerting settings according to your needs.

That’s it! Now a SIGNL4 alert is triggered every time Zabbix sends an alert to your Zabbix user.

Back-channel configuration for status updates

In the SIGNL4 web portal go to “Integrations” -> “Gallery” and look for the “Zabbix ()” integration. Note the arrow pointing to the left.

As “Zabbix URL” enter your Zabbix URL, e.g. https://your-zabbix-server/

Next, enter “Your Zabbix API token.” You can find this one as described here.

There’s no need for a username and password – just use the API token.

Enable the integration and click “Install.”

That’s it! Status updates are now sent from SIGNL4 to Zabbix.

For more information, have a look at the integration guide.

Key benefits

24/7 Alerting and escalation – Critical Zabbix alerts reach the right people instantly via mobile app, push, SMS, or voice call. This includes escalation, ensuring nothing slips through the cracks.
On-call duty management – Calendar-based on-call scheduling and automated routing replaces manual escalation, helping teams sleep better and respond smarter.
Rich, mobile-first notifications – Alerts include key incident details, so engineers can act quickly without logging into dashboards first.
Team collaboration and acknowledgment tracking – Everyone sees who has picked up an alert, for full transparency and structures response.
Reduced MTTA/MTTR – Faster acknowledgment and resolution mean less downtime, fewer escalations, and more stable operations.

What once felt like a constant struggle with missed notifications has turned into a structured, reliable alerting process. By connecting Zabbix with SIGNL4, the team not only strengthened their incident response but also made on-call duty a lot less of a burden – and that might be the biggest win of all.

The post 24/7 Alerting and Two-Way Integration with Zabbix and SIGNL4 appeared first on Zabbix Blog.

Saving Time with a Custom Zabbix Agent Installer

2025-12-04 Rizqi Firmansyah

Post Syndicated from Rizqi Firmansyah original https://blog.zabbix.com/saving-time-with-a-custom-zabbix-agent-installer/31843/

When managing large-scale infrastructure, the process of installing monitoring agents is often repetitive and time-consuming. Administrators must log into each server, manually run installation commands, and configure the agent to connect to the Zabbix server. To address this issue, the Zabbix Agent Deployer custom module was created. This module enables the direct installation of Zabbix agents on multiple hosts from the Zabbix Web interface.

The features of the Zabbix Agent Deployer module include:

Bulk host list input using a CSV file.
The ability to automatically add hosts to Zabbix and remotely install the Zabbix Agent on the
associated hosts.
The ability to display installation log results directly within the module.

With this approach, administrators can add new hosts to the monitoring system faster and more efficiently.

Key use cases for the Zabbix Agent installer

The Zabbix Agent Deployer module enables several practical scenarios, including:

1. Faster provisioning for new servers – When adding a large number of servers, agents can be installed simultaneously without requiring a login to each machine.

2. Standardized installation – All agents are installed in the same way using a centralized script, reducing the risk of misconfiguration.

3. Easier additional provisioning – Provisioning new servers is easier for users because they don’t need to configure them directly on the server.

Getting started with the Zabbix Agent Deployer module

Solution overview architecture

To use this module, the main steps are:

1. Upload the custom module to the Zabbix frontend in the /usr/share/zabbix/modules/ directory.

2. Enable the module from the Administration → General → Modules page, and click the Scan Directory button. Locate the Zabbix agent deployer module and click Enabled.

3. Once activated, the Zabbix agent deployer module can be accessed in the Data Collection menu. Here’s a screenshot of the Zabbix agent deployer module.

4. Prepare a CSV file like the format below, or download a sample CSV from the module page.

With this CSV file, we will add two hosts to Zabbix to be monitored and automatically install the Zabbix agent on them.

5. Upload the CSV file to the Zabbix agent deployer module page and click Apply.

6. The Zabbix agent deployer module will handle the process of adding hosts to Zabbix and installing the Zabbix agent. The status can be seen as follows:

From the image above, server1 and server2 were successfully added to Zabbix, and the Zabbix agent installation was successful!

7. Check out the Zabbix hosts list page. Hosts will appear according to the uploaded CSV file.

Conclusion

The implementation of this custom Zabbix Agent installer extends Zabbix’s capabilities beyond its built-in functionality. The Zabbix Agent Deployer module enables a more efficient bulk host addition process, as all steps from adding hosts to Zabbix to installing the Zabbix agent can be integrated through a single page.

If you’re interested in implementing this, please contact us. Bangunindo is a premium Zabbix partner in Indonesia. We’re ready to help you design, implement, and optimize your Zabbix solution to suit your needs.

The post Saving Time with a Custom Zabbix Agent Installer appeared first on Zabbix Blog.

Aruba Central API Monitoring with Zabbix

2025-11-25 Tibor Volanszki

Post Syndicated from Tibor Volanszki original https://blog.zabbix.com/aruba-central-api-monitoring-with-zabbix/31370/

Aruba Central is a SaaS solution that allows you to manage your Enterprise Aruba network environment. Due to the increasing number of cloud migrations, we can expect that more and more Aruba customers will move their on-premise environment to it, which will also mean a change in their monitoring environment. In this article, I will show you how to switch to API- based monitoring using Aruba Central and Zabbix. All custom resources mentioned can be found in my repository.

Aruba Central’s API

Oauth 2.0 is used, so you can forget the simple token management. At the end it is great, but for monitoring purposes it is overkill. There is pretty good documentation (referred to later) regarding how you can generate your access token, but after two hours it expires so you need to continually refresh it. To do this, you must use a refresh token, which can help you to get a new access token AND a new refresh token.

Within two hours, use the latest refresh token to repeat this action again. At this point you can imagine that this is not something you can implement easily by using the Zabbix GUI only. Well, maybe with some javascript magic, but otherwise there is no native support for this logic at this point of time. So how can we do this? In short:

Generate your client credentials
Generate your first token
Schedule the token refresh for every two hours
Update your host macro via Zabbix API
Use the token in Zabbix HTTP agent checks
Monitor your environment based on JSONPath pre-processing

Initial steps within Aruba Central

To manage your API access, you need to launch your “HPE Aruba Networking Central” application, so do NOT look into your workspace modules – the “Personal API clients” menu is NOT what we are looking for. Turn off the “New Central” view – at this point the early access version is not so useful (hopefully it will change soon).

The first time you get there, you will not see any items, but under the “My Apps & Tokens” tab you can click the “Add Apps & Tokens” button and generate it. Technically, this is already enough to start to monitoring your network infrastructure, but within two hours it would stop. So the relevant data for us are the “Client ID” and “Client Secret.” Feel free to revoke the recently created token at the bottom area as we do not need it.

Record your credentials

For this article, I am using a simple file to store all the credentials, which will be sourced into a bash script. Please keep in mind that storing your sensitive credentials in a single file is a BAD practice! Your SECO/CISO would probably have a few words with you about it, so please consider a better approach. A more secure way would be to use some Key Vault solution (like Azure, AWS, Google, or Hashicorp). Anyway, let’s continue with this unsecure example:

#!/bin/bash

### ZABBIX VARS ###

# URL of your zabbix instance (assuming you do not use the "/zabbix" ending, if yes, then add it to the end)
zabbix_url="https://your.zabbix.instance.net"
# Your Zabbix API token. If you do not know how to get it, check the documentation.
zabbix_api_token="1234_your_zabbix_api_key_5678"
# Create a host with a macro, remain at the "Macros" tab, turn on debug mode, look for "[hostmacroid] =>"
zabbix_macro_id="12345"

### ARUBA VARS ###
# To find yours, go here and check "Table: Domain URLs for API Gateway Access"
base_url="YOUR_ARUBA_CENTRAL_BASE_URL"
# Click on your profile in the Central app and you will find it there: 32 char long hexa string
client_id="YOUR_CLIENT_ID"
# provided in the previous step
client_secret="YOUR_CLIENT_ID"
# provided in the previous step
customer_id="YOUR_CUSTOMER_ID"
# your login credential
account_username="YOUR_CENTRAL_LOGIN_USERNAME"
# your login credential
account_password="YOUR_CENTRAL_LOGIN_PASSWORD"
# to be populated later
csrftoken=""
session=""
auth_code=""

Get or refresh your token and update the Zabbix host macro

The next steps are based on the official Aruba documentation, which you can find here. Please remember that there are many ways to achieve our target – this is just one example and probably not the most optimal one. Feel free to change / improve it with your code in your preferred scripting language.

The below script assumes that the file containing the credentials (previous step) is named as “variables” and located in the folder named “central.”

Filename: aruba_central_token_new.sh

Purpose: To be used for first time token generation. Later, you only have to refresh your token with the script after this one.

Remarks: Aruba is limiting this API query set, so you can run it only ONCE every 30 minutes! If you made a typo somewhere, wait 30 minutes before your next attempt or tweak the result files.

#!/bin/bash

basedir=central
source $basedir/variables

curl -s --noproxy '*' -v --cookie-jar $basedir/cookie --location --request POST "$base_url/oauth2/authorize/central/api/login?client_id=$client_id" \
--header "Content-Type: application/json" \
--data-raw "{
    \"username\": \"$account_username\",
    \"password\": \"$account_password\"
}" > $basedir/result1.raw 2>&1

grep 'Added cookie' $basedir/result1.raw > $basedir/result1.filtered

csrftoken=$(grep csrftoken $basedir/result1.filtered | awk -F '"' '{print $2}')
session=$(grep session $basedir/result1.filtered | awk -F '"' '{print $2}')

curl -s --noproxy '*' --request POST "$base_url/oauth2/authorize/central/api?client_id=$client_id&response_type=code&scope=all" \
--header "Content-Type: application/json" \
--header "Cookie: session=$session" \
--header "X-CSRF-Token: $csrftoken" \
--data-raw "{
\"customer_id\": \"$customer_id\"
}" > $basedir/result2.raw

auth_code=$(cat $basedir/result2.raw | jq -r .auth_code)

curl -s --noproxy '*' --request POST "$base_url/oauth2/token" \
--header "Content-Type: application/json" \
--data "{
    \"client_id\": \"${client_id}\",
    \"client_secret\": \"${client_secret}\",
    \"grant_type\": \"authorization_code\",
    \"code\": \"${auth_code}\"         
}" > $basedir/result3.raw

refresh_token=$(cat $basedir/result3.raw | jq -r .refresh_token)
access_token=$(cat $basedir/result3.raw | jq -r .access_token)

if [ "$refresh_token" == "null" ]; then
    echo "something went wrong... exiting now"
    exit 1
fi

echo $access_token > $basedir/token_access.latest
echo $refresh_token > $basedir/token_refresh.latest

echo "access_token: $access_token"
echo "refresh_token: $refresh_token"

curl -s --request POST \
--url "$zabbix_url/api_jsonrpc.php" \
--header "Authorization: Bearer $zabbix_api_token" \
--header "Content-Type: application/json-rpc" \
--data "{\"jsonrpc\": \"2.0\",\"method\": \"usermacro.update\",\"params\": {\"hostmacroid\": \"${zabbix_macro_id}\",\"value\": \"${access_token_new}\"},\"id\": 1}"

rm -f $basedir/cookie

Filename: aruba_central_token_refresh.sh

Purpose: To refresh your existing token. It is expecting an existing refresh token in the “token_refresh.latest” file, so better to run the previous script one time before this.

Remarks: You can run this script as many times you want, but it will result in new tokens only once per every two hours (when the current one expires). Therefore, refreshing too frequently is pointless.

#!/bin/bash

basedir=central
source $basedir/variables

refresh_token_current=$(cat $basedir/token_refresh.latest | tr -d '\n')
refresh_token_new=""

curl -s --noproxy '*' --request POST "$base_url/oauth2/token?client_id=$client_id&client_secret=$client_secret&grant_type=refresh_token&refresh_token=$refresh_token_current" > $basedir/result4.raw

refresh_token_new=$(cat $basedir/result4.raw | jq -r .refresh_token)
access_token_new=$(cat $basedir/result4.raw | jq -r .access_token)
expires_in=$(cat $basedir/result4.raw | jq -r .expires_in)

if [ "$refresh_token_new" == "null" ]; then
    echo "something went wrong... exiting now"
    exit 1
fi

echo $access_token_new > $basedir/token_access.latest
echo $refresh_token_new > $basedir/token_refresh.latest

echo "access_token: $access_token_new"
echo "refresh_token: $refresh_token_new"
echo "expires_in: $expires_in"

curl -s --request POST \
--url "$zabbix_url/api_jsonrpc.php" \
--header "Authorization: Bearer $zabbix_api_token" \
--header "Content-Type: application/json-rpc" \
--data "{\"jsonrpc\": \"2.0\",\"method\": \"usermacro.update\",\"params\": {\"hostmacroid\": \"${zabbix_macro_id}\",\"value\": \"${access_token_new}\"},\"id\": 1}"

In my case, both the scripts and variables files are in the same “central” folder, which is in a git repository. Each time I call one of the scripts, it will record the new tokens in files, which are committed and pushed to the repo. In my own implementation, this is how I call the refresh script and sync the result with my repo:

git checkout master

basedir=central
source $basedir/variables
bash $basedir/aruba_central_token_refresh.sh

git add .
git commit -m "save the new tokens"
git push origin master

Schedule your token management

You must run your refresh script at least once per every two hours. To make this happen you have many options, including:

cron (old-school, outdated way)
systemctl timer (a better way, but only if it is monitored)
Jenkins / Github Actions/etc.
Zabbix itself, by calling your bash script

In my case, Jenkins does the scheduling and execution and the job is monitored via Zabbix.

Monitor your network infrastructure

When everything is in place, then the monitoring part is pretty simple. The usual JSONPath based logic can be used. API call documentation can be found here. The template contains only the wireless components, since I do not have my switches in Central. Implementing the switching part should not be difficult – just have a look at the “Switch” section, then clone and adjust one of your “get” items.

Screenshots

Latest data – tag based filtering:

Latest data – Site health

Latest data – Gateway info

Latest data – AP info

Triggers:

Some triggers are intentionally disabled, because they are a bit redundant. However, I wanted to cover all options. Sometimes less alerting is better if you have a ticketing system integration, otherwise your monitoring system will turn into a ticket factory.

Known issues and limitations

Since we are not querying the devices directly, some delay can be expected. Based on my recent testing, the delay compared to real time is between 3-10 minutes. In my test I disconnected my test environment and then started to do manual updates frequently. Some items got the real state earlier, some only later.

If your refresh script will malfunction for whatever reason (normally it should not), then you may have to run the other script once to generate a new token, or you can go to the GUI and check the last refresh token, with which you can override the content of the “token_refresh.latest” file.

Aruba is limiting the number of API queries to 5,000 per day. This could seem annoying, but it is way more than what you need (you should expect less than 1,000 in normal conditions, depending on your update frequency).

Zabbix API will not authorize your call unless you insert a line into your apache vhost configuration. This is a more generic Zabbix API issue that is not related to Aruba Central.

SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1

If Aruba Central has a maintenance activity, then the token refreshing way could break. Running the token request script once should address the issue.

Summary

Aruba Central’s API is pretty decent, but if you start from zero it could take a while to get to the end of it. With this guide, my intention was to speed you up, but please do not consider my scripts and the shown example as the only or best possible way – I’m just hoping it can give you a good base for your own solution. Have fun!

The post Aruba Central API Monitoring with Zabbix appeared first on Zabbix Blog.

Making PaperCut NG Observable with Zabbix

2025-11-18 Patrik Uytterhoeven

Post Syndicated from Patrik Uytterhoeven original https://blog.zabbix.com/making-papercut-ng-observable-with-zabbix/31244/

In most organizations, printing is an essential but often invisible service. When it works, nobody notices. When it fails, productivity stalls. That’s why monitoring your print environment is just as important as monitoring servers, databases, or network devices.

At Opensource ICT Solutions, we specialize in turning complex systems into observable services. One recent example is our integration of PaperCut NG with Zabbix. This allows IT teams to track the health of their print infrastructure in real-time — everything from server resources to individual printers and devices.

Why monitoring PaperCut matters

PaperCut NG does much more than queue print jobs. It enforces quotas, integrates with authentication systems, and manages fleets of devices. If the database runs out of connections, the disk fills up, or the license expires, users feel the impact instantly.

By integrating PaperCut with Zabbix, we make these risks visible long before they become business problems. The result is:

Proactive detection of printer errors, low toner, or license issues.
Capacity planning through trend analysis of disk usage, memory, and DB connections.
Unified visibility — PaperCut health checks appear right alongside servers, networks, and applications in Zabbix dashboards.

How the integration works

The magic happens through the PaperCut System Health API and Zabbix’s flexible data collection methods.

HTTP agent items

Zabbix fetches raw JSON data directly from PaperCut using an HTTP agent item, such as:

This single call provides a full snapshot of server health.

Dependent items + JSONPATH

Instead of hammering the API with multiple requests, we extract the needed fields using dependent items with JSONPATH preprocessing.

For example:

This design means one request can populate dozens of metrics, keeping monitoring both efficient and lightweight.

Calculated items

Some values aren’t directly available from PaperCut. In those cases, we create calculated items inside Zabbix.

For example, the percentage of active DB connections is derived as:

This allows us to set intelligent triggers like “DB connections > 90%” without requiring PaperCut to calculate it for us.

Low-level discovery (LLD) for devices and printers

Perhaps the most powerful part of this integration is automatic discovery.

Printer LLD → Queries /api/health/printers and creates items and triggers per printer. If a printer goes into Paper Jam or No Toner, Zabbix knows immediately.
Device LLD → Queries /api/health/devices and builds items dynamically for each discovered device, tracking states like OK, WARNING, or ERROR.

This ensures that new printers and devices are monitored automatically — no manual configuration required!

Why this matters

Bringing all of this together, the integration turns PaperCut NG into a fully observable service inside Zabbix.

Efficiency → One API call, dozens of metrics.
Scalability → Automatic discovery of printers and devices.
Robustness → Alerts and dashboards for licenses, resources, and print queues.

For IT teams, this means fewer surprises, faster troubleshooting, and more confidence in a service that often goes unnoticed until it fails.

Our expertise

This PaperCut integration is just one example of how we at Opensource ICT Solutions help organizations unlock the full potential of Zabbix. We don’t just install monitoring – we design intelligent, scalable integrations that make hidden systems visible. Whether it’s print management, databases, custom applications, or network devices, we know how to extend Zabbix to fit your environment and give you the insights that matter most.

Feel free to download our template and documentation for free from our GitHub: https://github.com/OpensourceICTSolutions/ZabbixPapercutNG

Want to make your business-critical systems truly observable? Let’s talk about how we can tailor Zabbix to your needs: [email protected]

The post Making PaperCut NG Observable with Zabbix appeared first on Zabbix Blog.

Monitoring Website Changes with Zabbix Browser Item

2025-11-11 Adi Rusmanto

Post Syndicated from Adi Rusmanto original https://blog.zabbix.com/monitoring-website-changes-with-zabbix-browser-item/31684/

In today’s digital era, information is an asset and most of it is obtained from websites. The ability to automatically monitor website content changes has become a crucial competitive advantage, as even small changes on a website can affect business strategies, security postures, and data-driven decision-making. Accordingly, Zabbix 7.0 saw the introduction of a new feature called Browser Item, which allowed users to perform advanced website monitoring using a browser.

The Browser Item feature includes the ability to:

● Capture screenshots of the current website state
● Measure website performance and availability metrics
● Extract and analyze data from web pages
● Generate automatic alerts based on detected changes or errors

This means Zabbix is no longer limited to traditional IT infrastructure monitoring. It can now also serve as a tool for monitoring strategic external information.

Key use cases for website change monitoring with Zabbix

The Zabbix Browser Item opens up many valuable use cases for organizations that want to proactively track website changes. Below are some key examples:

Monitoring release notes

Tracking vendor release notes is essential for IT teams. With Zabbix, we can automatically detect new releases, extract relevant information, and notify the appropriate team members so they can respond faster.

Tracking security advisories

Security advisories are critical for maintaining a strong security posture. By monitoring websites that publish vulnerability information using Zabbix, security teams can be promptly alerted about new threats and take timely actions to reduce risks.

Monitoring competitor websites

In a competitive market, staying informed about competitor activities is vital. Zabbix allows users to monitor competitor websites for pricing updates, new product offerings, marketing campaigns, or news announcements, while providing valuable business intelligence to support strategic decisions.

Monitoring tender announcements

Zabbix can also monitor websites for new tender announcements from government portals or business partners, ensuring our organization stays aware of the latest business opportunities.

Ensuring internal website integrity

Beyond external sites, we can also use the Browser Item to ensure the integrity and availability of our own websites. It helps detect unexpected content changes, broken links, or performance degradation that may affect the user experience or signal potential issues. Proactive monitoring helps maintain a high-quality user experience and protect our brand reputation.

Getting started with website change monitoring in Zabbix

Solution overview architecture

This diagram shows how Zabbix uses a WebDriver to capture and analyze website content.
The collected data is stored in Zabbix for visualization and alerts when changes are
detected.

Step-by-step configuration

In this example, we’ll monitor changes on the Nginx Security Advisories webpage.

Step 1: Prepare the Web Driver

Zabbix requires a Web Driver to perform browser-based monitoring. One commonly used option is Selenium, which can be deployed using the following Docker image:

https://hub.docker.com/r/selenium/standalone-chrome

Step 2: Configure WebDriverURL on Zabbix server or proxy

Update the WebDriverURL parameter in your Zabbix Server or Zabbix Proxy configuration to point to the Selenium service you deployed.

Step 3: Create a Browser Item in Zabbix

1. Create a host if it doesn’t already exist.

2. Add a new item with the following settings:

Type: Browser
Type of information: Text

The key part is the script section. Below is the example script.

The script uses two methods:

browser.navigate method defines the URL to be monitored
browser.findElements method specifies the page section where changes should be detected

Note: The StartBrowserPollers parameter must be enabled on the Zabbix server or proxy configuration for browser items to work. It is enabled by default with the value StartBrowserPollers=1.

Step 4: Create dependent items

The Browser Item produces a JSON result containing website data. This item serves as the master item for dependent items such as:

Extracting the latest security advisories
Capturing a website screenshot

Step 5: Create a trigger for change alerts

Create a trigger that compares the current and previous values of the “latest security advisories” item. If any change is detected, Zabbix will automatically send an alert notifying your team of the update.

Step 6: Display data on the dashboard

To visualize the monitored data, we can use the Item History widget on a Zabbix dashboard to show both the latest security advisories and the corresponding screenshot, for example.

Conclusion

The Browser Item feature in Zabbix 7.0 elevates website monitoring beyond simple availability checks. It enables comprehensive monitoring of website changes, unlocking a variety of use cases such as tracking release notes, security advisories, competitor activity, and more.

If you’re interested in implementing this capability, feel free to contact us. Bangunindo is a Zabbix Premium Partner in Indonesia, ready to help you design, implement, and optimize your Zabbix monitoring solution to fit your specific needs.

The post Monitoring Website Changes with Zabbix Browser Item appeared first on Zabbix Blog.

Monitoring a Starlink Dish with Zabbix

2025-10-21 Alexander Petrov-Gavrilov

Post Syndicated from Alexander Petrov-Gavrilov original https://blog.zabbix.com/monitoring-a-starlink-dish-with-zabbix/31543/

Did you realize that you can monitor a Starlink dish using just Zabbix? The idea (or rather the need) to use Starlink came to me almost as soon as I moved to a fairly rural area. Local internet providers have not yet “provided” fiberoptic or stable mobile connectivity to places like this, and while searching for a solution I accidentally discovered that Starlink was already providing service to some local companies. As I later found out, they also offered service in my area for residential customers.

To make a long story short, since internet access is crucial in the IT field, I decided to acquire and then monitor my very own Starlink dish. At first, this proved challenging because regular user data access is quite limited. However, thanks to Zabbix browser monitoring, I managed to solve it fairly easily. In this post I will share my solution with you, including the template.

Table of Contents

Monitoring configuration

First, you need to make sure you have Zabbix installed (either a Zabbix proxy or server) on the same network that the Starlink dish and router are on. The next step is to configure Zabbix for browser monitoring.

WebDriver installation

# podman run --name webdriver -d \
-p 4444:4444 \ 
-p 7900:7900 \
--shm-size="2g" \
--restart=always -d docker.io/selenium/standalone-chrome:latest

Port 4444 will be the port on which the WebDriver will be listening, and port 7900 will be used by NoVNC, which allows us to observe browser behavior in case a browser with a GUI is used.

Zabbix server/proxy configuration

After WebDriver is installed, we need to set up the communication between Zabbix and the driver. This can be done by editing the Zabbix server/proxy configuration file and updating the following parameters:

### Option: WebDriverURL 
# WebDriver interface HTTP[S] URL. For example http://localhost:4444 used with 
# Selenium WebDriver standalone server. 
# 
# WebDriverURL= 
WebDriverURL=http://localhost:4444 
### Option: StartBrowserPollers 
# Number of pre-forked instances of browser item pollers. 
# 
# Range: 0-1000 
# StartBrowserPollers=1 
StartBrowserPollers=5

With the configuration parameters in place, restart the Zabbix server/proxy to apply the changes:

systemctl restart zabbix-server

Creating a host

First, we need to navigate to the “Data collection” > “Hosts” section and create a host that represents our Starlink dish. The host in my example will look like this:

The host also has a user macro:

{$LINK} with value: http://webapp.starlink.com to point to the correct Starlink dish web app:

Creating a browser item

We will now configure our browser item to collect and monitor the list of metrics exposed in the Starlink browser app:

We are using the bare minimum here, so make sure the update intervals are as frequent as you need. However, I would not recommend updating it more frequently than every 5 minutes. It’s also not a good idea to store the history, since it is already stored trough dependent items.

The most important part of the item is the script itself:

var browser, result;
var opts = Browser.chromeOptions();

opts.capabilities.alwaysMatch['goog:chromeOptions'].args = [];
browser = new Browser(opts);
browser.setScreenSize(Number(1980), Number(1020));

try {
    var params = JSON.parse(value);
    browser.navigate(params.url);

 // Wait for the dish to report status
    Zabbix.sleep(2000);

    // Find the JSON text element(s)
    var jsonElements = browser.findElements("xpath", "//div[@id='root']/div[@class='App']/div[@class='Main']/div[2]/div[@class='Section'][2]/pre[@class='Json-Format']/div[@class='Json-Text']");
    var extractedData = [];

    for (var i = 0; i < jsonElements.length; i++) {
        var text = jsonElements[i].getText();

        // Try parsing JSON
        try {
            extractedData.push(JSON.parse(text));
        } catch (e) {
            // If not valid JSON, include raw text instead
            extractedData.push({ raw: text, error: "Invalid JSON format" });
        }
    }

    // Collect result 
    result = browser.getResult();

    // Replace with parsed JSON data
    result.extractedJsonData = extractedData.length === 1 ? extractedData[0] : extractedData;

}
catch (err) {
    if (!(err instanceof BrowserError)) {
        browser.setError(err.message);
    }
    result = browser.getResult();
}
finally {
    // Return a clean JSON object
    return JSON.stringify(result.extractedJsonData);
}

So what does this script do? It opens the Starlink web app, waits for the Starlink dish to output all the status data, and, after a bit of parsing, returns the data highlighted in the screenshot:

Now we can click on the three dots on the left of our newly created item in the items page and proceed to create dependent items for each value we are interested in!

Creating dependent items

Now we just click here:

As an example, to create an item that monitors the hardware version we can create an item like this:

With JSONPath preprocessing:

In the end we get the data in Zabbix:

All other items (except alerts) will follow the same logic – just update the item name, key, and JSONPath in preprocessing to extract the required values.

Creating dependent LLD item prototypes

To automate the alerts items creation, we can create a dependent discovery rule. In the “Discovery” section, create a new discovery rule:

With preprocessing using Java Script:

var data = JSON.parse(value);
var alerts = data.alerts;
var lld = [];

for (var key in alerts) {
    if (alerts.hasOwnProperty(key)) {
        lld.push({
            "{#ALERT}": key
        });
    }
}

return JSON.stringify({ data: lld });

This will provide us with following JSON data:

{
  "data": [
    {
      "{#ALERT}": "dishIsHeating"
    },
    {
      "{#ALERT}": "dishThermalThrottle"
    },
    {
      "{#ALERT}": "dishThermalShutdown"
    },
    {
      "{#ALERT}": "powerSupplyThermalThrottle"
    },
    {
      "{#ALERT}": "motorsStuck"
    },
    {
      "{#ALERT}": "mastNotNearVertical"
    },
    {
      "{#ALERT}": "slowEthernetSpeeds"
    },
    {
      "{#ALERT}": "softwareInstallPending"
    },
    {
      "{#ALERT}": "movingTooFastForPolicy"
    },
    {
      "{#ALERT}": "obstructed"
    }
  ]
}

All that’s left ‘to do is to create a dependent item prototype:

With preprocessing, of course:

JSONPath will transform to extract each specific alert and “Boolean to Decimal” will save us some space in the database by tranforming true/false booleans to digits.

Result

In the end, we can monitor all the data:

Even more data can be collected using exporters – if you are willing to do a bit of extra configuration, of course! Let me know if you are interested, and I will show you a completely different approach with a template.

Before I forget, the template used in this tutorial can be found here.

The post Monitoring a Starlink Dish with Zabbix appeared first on Zabbix Blog.

Running Zabbix with MariaDB and Galera Active/Active Clustering

2025-09-30 Nathan Liefting

Post Syndicated from Nathan Liefting original https://blog.zabbix.com/running-zabbix-with-mariadb-and-galera-active-active-clustering/31104/

High availability on a platform like Zabbix is a hard requirement for many users. With native high availability on the Zabbix servers, proxies, and at the frontend through various solutions for web servers, all that’s left is at the database layer. Any downtime in your MariaDB database would disrupt your monitoring availability, at the least on the frontend side of things in case of proxy buffering. Let’s have a look at the easiest way to create a high availability (HA) architecture for Zabbix using MariaDB with built-in Galera clustering – by removing single points of failure from your database and finalizing the HA puzzle for Zabbix.

Architecture overview

Let’s start of with the MariaDB + Galera number one design requirement. For a proper quorum to be made, 3 nodes should be used in the cluster. With only two nodes in a Galera cluster, quorum rules become a bit of a headache, as Galera uses a majority vote (more than half the nodes) to decide if the cluster can still accept writes. In a two-node setup, all is good when the database is online. But when we lose one node, quorum is lost and that node needs to rejoin.

This makes a two-node setup fragile but not impossible, and it does work with Zabbix since we do only have one Zabbix server active at the time. In a split-brain scenario where both nodes either think they are the last to leave, you might have to decide which node you think has your up-to-date data. We will detail both scenario’s, but the principle remains the same. We will use MariaDB as our database and Galera will be used to create a primary/primary cluster. In such a cluster, all nodes in the cluster are writeable, which is great for the Zabbix native HA.

When we look in the Zabbix database, we can see that Zabbix keeps all of it’s Zabbix server HA information and states in the database.

This means that whatever one Zabbix server node writes into the database will also be replicated to all other nodes in the MariaDB Galera cluster.

The design

Knowing what we know now, we can create a very simple design for a solid Zabbix HA setup with Mariadb + Galera. When we have a single Zabbix frontend and we keep to the MariaDB + Galera requirement of having 3 database nodes, we get a fairly simple setup, as seen below.

In this setup, each Zabbix server connects to its own Database node and we don’t need added complexity by using load balancers. However, we do get an automatic failover from the Zabbix servers, as they know exactly which node is active through the database. However, in this situation we are still left with 3 frontends that do not have automatic failover, simply because we do not have database aware Apache or NGINX. This also works in a two database setup, with the side note that you might have quorum issues to manually resolve after an outage:

Adding onto this setup, we could install a VIP, load balancer, or something like HA proxy in front of the frontend to make a failover happen there as well. Keep in mind though, the failover needs to happen based on whether or not the webfrontend can reach a writeable database.

Optional Arbitrator

If you are set on running only 2 database nodes (your wallet is thankful), but still worried about quorums, we can bring in the ARBITRATOR.

If there are only 2 Database nodes in your Galera cluster, not to worry! It’s definitely possible even while maintaining a good quorum resolution in case of outages.

What about load balancing?

Lastly, it is also possible to add load balancing to the mix. Let’s say, for example, you cannot add a VIP to your environment but still need your WEB servers to failover. A load balancer can provide the solution here.

We still prefer to run the Zabbix servers with a direct database connection, but even there a load balancer could be added if you wish. However, please keep in mind that the more load balancers you add, the more complex troubleshooting might become. The whole idea about the setup without load balancers is to have a solid Zabbix setup that is easy to maintain, while providing high availability.

Conclusion

In the end, even with a minimal setup of 2 DB nodes, 2 Zabbix servers, and 2 WEB frontends, we can make a high availability setup. As we’ve shown with Galera, this setup becomes highly flexible, allowing us to run without automatic WEB failover all the way up to including complicated load balancers.

High availability doesn’t have to be overly complicated in a setup like this – it really is all about how far you want to push things. Besides that, in this setup everything is horizontally scalable on the database side. Do keep in mind, however, that Zabbix does still run in an Active/Passive setup.

I hope you enjoyed reading this blog post. If you have any questions or need help configuring anything in your Zabbix setup feel free to contact me and the team at Opensource ICT Solutions. We build a ton of cool stuff like this and more!

Nathan Liefting

https://oicts.com

A close up of a logo Description automatically generated

The post Running Zabbix with MariaDB and Galera Active/Active Clustering appeared first on Zabbix Blog.

Building HA Zabbix with PostgreSQL and Patroni

2025-09-16 Patrik Uytterhoeven

Post Syndicated from Patrik Uytterhoeven original https://blog.zabbix.com/building-ha-zabbix-with-postgresql-and-patroni/30960/

Running a monitoring platform like Zabbix in a production environment demands reliability and resilience. When your monitoring solution is down, you’re flying blind – and for many organizations, that simply isn’t acceptable. This post introduces a robust high-availability (HA) architecture for Zabbix, using PostgreSQL, Patroni, etcd, HAProxy, keepalived and PgBackRest. Built on RHEL 9 or derrivates, this solution combines modern open-source tools to provide automatic failover, load balancing, and seamless monitoring, all while maintaining consistency and performance.

Architecture overview

The HA design consists of multiple layers working in tandem to maintain continuity even during node or service failures:

Database Cluster Layer

2 or more nodes form the PostgreSQL cluster, managed by Patroni and coordinated using etcd. At any given time, one node is the primary (read/write), and the others are hot standbys ready to take over automatically.

Consensus layer

etcd runs on the same nodes and acts as the distributed configuration store and coordination layer for Patroni. It ensures a consistent cluster state and enables safe failover decisions.

Load balancing layer

Two HAProxy nodes provide a single point of entry for all clients (including Zabbix), routing requests to the current PostgreSQL primary. These nodes are monitored and coordinated via Keepalived to maintain a floating Virtual IP (VIP), ensuring seamless failover at the connection layer.

Backup layer

A separate backup server is responsible for running PgBackRest, which handles full and incremental backups, WAL archiving, and Point-In-Time Recovery (PITR). This server communicates securely with all database nodes over SSH.

Monitoring layer

Two Zabbix servers, running in active-passive mode, continuously monitor all layers of this stack including the HAProxy health, Patroni cluster role, and etcd status by accessing the PostgreSQL VIP for backend connectivity.

This multi-tiered setup ensures that no single failure be it a database, load balancer, or monitoring server brings down the monitoring platform.

Why HA matters for Zabbix

Zabbix depends heavily on its PostgreSQL database backend. Every metric, trigger, event, and alert is stored there. If PostgreSQL becomes unavailable, even briefly, data loss or monitoring blind spots can occur. That’s why introducing HA at the database layer is a crucial step when scaling Zabbix for enterprise environments.

While Zabbix itself supports HA at the application level, this architecture ensures that the database backend is also fully fault-tolerant, using modern consensus-based clustering with automatic failover.

Component overview

To achieve HA, we bring together several specialized components, each fulfilling a critical role in the system:

PostgreSQL

The relational database engine used by Zabbix. In this example setup, it runs on three nodes, forming a cluster managed by Patroni.

Patroni

Patroni is the orchestrator for the PostgreSQL cluster. It monitors node health, manages replication, promotes standbys when needed, and ensures only one writable leader exists at any time. Patroni leverages a distributed consensus store in this case, etcd but other DCS’s are possible to coordinate decisions across the cluster.

etcd

etcd is a lightweight and highly available key-value store used by Patroni to maintain the cluster’s state. It stores leader election data, health statuses, and locks. We deploy it as a three-node cluster, co-located with the PostgreSQL nodes for convenience, though this setup can be scaled independently if needed as etcd is very latency prone.

HAProxy

To simplify application connectivity, HAProxy acts as a load balancer in front of the database cluster. It monitors the role of each node using Patroni’s REST API and routes connections to the active primary server. If the leader fails, HAProxy automatically reroutes traffic to the new primary.

Keepalived

Keepalived provides a floating virtual IP address (VIP) across the HAProxy nodes. This VIP allows client systems, such as the Zabbix frontend, to connect to a single stable IP even if one HAProxy node fails.

PgBackRest

To protect the data itself, we use PgBackRest for full and incremental backups, as well as Point-In-Time Recovery (PITR). A dedicated backup server is included to pull and store archive logs and backups securely via SSH.

Zabbix server

Finally, we run two Zabbix servers in active-passive mode. Both are configured to connect to the PostgreSQL cluster through the VIP exposed by HAProxy. The Zabbix frontend is deployed on both nodes as well, ensuring continued accessibility through the load-balanced setup.

Topology at a glance

Here’s a simplified view of the architecture:

2 or more database nodes (PostgreSQL + Patroni + etcd)
Two HAProxy nodes, each configured with Keepalived to manage a floating virtual IP
One backup node for PgBackRest
Two Zabbix servers pointing to the PostgreSQL VIP

All systems are tied together with consistent hostname mappings, time synchronization (Chrony), and service monitoring.

Notes:

PgBackRest is directly connected to all three PostgreSQL nodes, allowing it to archive WAL segments and pull backups regardless of which node is primary.
This design enables full standby backups and supports Point-In-Time Recovery (PITR).
HAProxy ensures Zabbix always talks to the current primary node, while Patroni and etcd handle automatic failover and cluster state management.

Design rationale

This setup prioritizes resilience and self-healing. If any single component fails a database node, a load balancer, or even a monitoring server the system continues to function.

Using Patroni with etcd ensures that failovers are handled automatically, without human intervention. HAProxy ensures client traffic is always routed to the current primary, while Keepalived ensures that this routing layer itself is highly available.

We opted for PgBackRest over simple scripts or base backups because it provides not just efficient incremental backups, but also full WAL archiving and point-in-time recovery, which are invaluable for both disaster recovery and debugging.

Lastly, we chose to integrate Zabbix itself into this HA design, treating it not just as a application but as a fully resilient service able to monitor itself, so to speak.

Real-world considerations

Resource planning: While our nodes run comfortably, scaling this setup to heavy workloads requires careful tuning of memory, I/O, and PostgreSQL parameters.
etcd placement: Although we run etcd co-located with the database nodes in this example, separating etcd onto dedicated infrastructure is ideal for large-scale environments. This avoids resource contention and preserves quorum in extreme failure scenarios.
Monitoring the monitors: Zabbix itself must be monitored. In our setup, each component including etcd, Patroni, and PostgreSQL exposes health endpoints that can be used by Zabbix agents or scripts to generate alerts on replication lag, cluster health, and failover events.

Conclusion

This architecture provides a solid foundation for running Zabbix in a fault-tolerant, production-ready environment. It not only ensures high availability for the database layer but also offers flexibility, observability, and operational safety.

Whether you’re running internal infrastructure monitoring or offering Zabbix as a managed service, adopting this type of HA setup removes single points of failure and gives you peace of mind — all using open-source technologies that are battle-tested and widely supported.

If you need assistance with the migration or want to ensure best practices for scaling and optimizing Zabbix, don’t hesitate to reach out to OICTS. We are a Zabbix Premium Partner operating globally, with offices in the USA, UK, Netherlands, and Belgium, and we’re ready to help you every step of the way.

The post Building HA Zabbix with PostgreSQL and Patroni appeared first on Zabbix Blog.

Revolutionizing Zabbix Maintenance with Artificial Intelligence

2025-09-11 Grover Taipe

Post Syndicated from Grover Taipe original https://blog.zabbix.com/revolutionizing-zabbix-maintenance-with-artificial-intelligence/31284/

Can you imagine being able to schedule maintenance in Zabbix by simply telling a program: “I need to put the web server in maintenance tomorrow from 8 to 10 with ticket 100-178306”? That’s exactly what the Artificial Intelligence (AI) Scheduler Zabbix project I’ve developed does!

What problem does it solve?

Anyone who has worked with Zabbix knows that scheduling maintenance can sometimes be tedious, especially when you need to:

Configure complex routine maintenance
Handle Zabbix API bitmasks for specific days of the week or month
Search for specific hosts or groups
Document associated tickets

This project eliminates that friction by allowing the use of natural language to create both one-time and routine maintenance.

The magic behind the code

Conversational artificial intelligence

The system integrates both OpenAI GPT-4 and Google Gemini to interpret natural language requests. The AI doesn’t just understand what you want to do, but automatically:

Detects servers, groups, and dates
Identifies ticket numbers (XXX-XXXXXX format)
Automatically calculates complex Zabbix bitmasks
Generates contextual responses with examples

Fig. 1. Adding the AI Scheduler widget to your Zabbix dashboard

Advanced routine maintenance

What really stands out is its ability to handle complex patterns. Here are some practical examples that work:

“Daily backup for srv-backup from 2 to 4 AM with ticket 200-8341 until February 2027”
“Thursday and Friday maintenance from 5 to 7 AM until January 2027”
“Cleanup on the first Sunday of each month with ticket 100-178306 until December 2026”

Fig. 2. AI-generated maintenance summary with all calculated parameters

Elegant architecture

The project uses a three-layer architecture:

Frontend: Custom widget for Zabbix
Backend: Flask API with AI integration
Zabbix: Native API to create maintenance

Fig. 3. Maintenance successfully created and visible in Zabbix interface

Super-simple installation

One of the best features is how easy it is to get it running:

cp .env.example .env

You only need to configure your Zabbix URL and AI API key:

 docker compose up -d --build

And that’s it! You have an AI assistant working.

Multi-instance support

For organizations with multiple Zabbix servers, the project includes configuration for up to 5 simultaneous instances, each with its own configuration.

What impresses me most

Intelligent date detection

The system understands natural expressions like:

“Tomorrow from 8 to 10” → Next date with specific schedule
“Sunday from 2 to 4 AM” → Next Sunday at those hours
“24/08/25 10:00am” → Automatically converts the format

Automatic Bitmask management

Zabbix API bitmasks can be notoriously complicated. This system calculates them automatically:

Thursday and Friday = 8 + 16 = 24
Sundays only = 64
First week of the month with specific configuration

Fig. 4. Complex weekly maintenance scheduling with automatic bitmask calculation

Why is it important?

This project represents a natural evolution in systems administration. Instead of memorizing complex syntax or navigating multiple menus, you simply describe what you need in natural language. It’s especially valuable for:

Operations teams handling multiple maintenance tasks
Companies that need to document associated tickets
Organizations with complex maintenance patterns

The future is here

Projects like this demonstrate how artificial intelligence can make complex technical tools more accessible without sacrificing functionality. It’s not just automation – it’s intelligence applied to real infrastructure problems. If you work with Zabbix and are tired of manually configuring maintenance, this project is definitely worth checking out. It’s open source, well documented, and solves a real problem that many of us face every day. You can find the complete project on GitHub.

The post Revolutionizing Zabbix Maintenance with Artificial Intelligence appeared first on Zabbix Blog.

Migrating from PRTG to Zabbix: A High-Level Guide

2025-09-02 Patrik Uytterhoeven

Post Syndicated from Patrik Uytterhoeven original https://blog.zabbix.com/migrating-from-prtg-to-zabbix-a-high-level-guide/30845/

For companies looking to migrate from PRTG Network Monitor to Zabbix, one of the most critical aspects is making sure a smooth migration of monitored devices and configurations. While there is no official tool to directly migrate between the two platforms, creating a bridge using custom export/import scripts allows for an effective and large migation. This blog post outlines a practical approach to achieving that migration based on the export/import methodology we at Opensource ICT Solutions previously implemented for one of our clients.

Why migrate?

While PRTG offers an intuitive interface and is popular for its ease of use, Zabbix provides:

Greater flexibility and scalability
Full open-source licensing
More powerful automation and templating
A robust API for integrations
Lower costs, especially since Paessler was sold to an investor

These features make Zabbix an attractive choice for teams looking to scale or standardize on open-source infrastructure.

Migration overview

The migration involves two key steps:

Exporting PRTG device information
Importing data into Zabbix

Because the two systems are conceptually and structurally different, we focused our scripts on migrating what is most transferable: device names, IP addresses, and interface types. SNMP versions or PRTG-specific sensor details were excluded or simplified where not applicable to Zabbix. PRTG, for example, will only export probes that have an OID that was not built-in in PRTG but added later, making our export incomplete. This does not mean we did a partial migration, it just means we have not included it in the automated approach.

Step 1: Exporting from PRTG

We developed a Python-based script that interacts with the PRTG API to extract monitored device data and export it to a CSV file. The script filters out irrelevant objects and organizes the output for easy Zabbix processing.

This creates a clean CSV, like this:

Device Name, IP Address, Interface Type
zabbix-server,10.0.0.10,agent
ServerA,192.168.0.2,SNMP
ServerA,192.168.0.2,agent
core-switch,192.168.0.1,SNMP

This file serves as a clean, structured inventory of monitored devices.

Note: SNMP version fields were excluded in the final export, as Zabbix does not currently display or rely on an SNMP version in the same way PRTG does.

Step 2: Importing into Zabbix

Using Zabbix’s API, we created an import script that reads the CSV and:

Creates host entries
Assigns them to the appropriate host group
Adds relevant interfaces (e.g., Agent,ILO,SNMP or a combination of …)

Each host is configured based on its detected interface type in PRTG.

On the Zabbix side, we used the Zabbix API to automate the creation of hosts, interfaces, and template assignment. The import script reads the CSV line-by-line and takes action based on the interface type.

Considerations and “gotchas”

Templates: We didn’t add templates, as there is no 1:1 solution – PRTG has a different concept and adding a standard template would be possible but probably not the best solution.
Host Groups: For ease of use and the limited time we had, we added all hosts in a temporary host group made for the migration. Although we do have scripts that take it out from PRTG and create it in Zabbix, in this particular migration it was not needed.
Permissions: The API token used in the import script must have sufficient privileges to create hosts.

What is NOT migrated

Because of fundamental differences between the platforms, the following are not directly migrated:

Historical data or sensor readings: Mainly because the customer had no hard requirement for it.
Custom PRTG notifications or dependencies: It was easier to manually re-create them.
Maps or dashboards: The Zabbix approach is so different that it was easier to recreate it manually (and improve).
Sensors: Zabbix is working with a different concept.

Post-migration tips

Validation: After the import, verify that each host is reachable and monitored correctly in Zabbix.
Discovery: Consider using Zabbix’s LLD (Low-Level Discovery) to dynamically find interfaces, disks, or other entities.
Housekeeping: Disable PRTG monitoring only after confirming Zabbix is fully operational.

Conclusion

Migrating from PRTG to Zabbix is not a one click operation, but with some scripting, planning, and experience from a partner like us, it can be done efficiently and with minimal disruption. The custom export/import scripts act as a reliable bridge between the two systems, allowing for a clean transfer of your monitoring inventory. From there, Zabbix’s automation and scalability features can help take your monitoring to the next level.

The post Migrating from PRTG to Zabbix: A High-Level Guide appeared first on Zabbix Blog.

Running Zabbix with PostgreSQL and PG Auto Failover

2025-08-12 Patrik Uytterhoeven

Post Syndicated from Patrik Uytterhoeven original https://blog.zabbix.com/running-zabbix-with-postgresql-and-pg-auto-failover/31026/

Running a monitoring platform like Zabbix in a production environment requires bulletproof availability at the database layer. Any downtime in PostgreSQL, even for seconds, can disrupt monitoring visibility, triggering blind spots in alerts and data collection.

This post introduces a streamlined High-Availability (HA) architecture for Zabbix using PostgreSQL, pg_auto_failover, HAProxy, and PgBackRest. Built on RHEL 9 or derivatives, this architecture removes single points of failure and automates failover using minimal external dependencies, making it a strong candidate for modern observability backends.

Architecture overview

This HA design simplifies deployment by using a dedicated monitor node to orchestrate automatic failover between two PostgreSQL database nodes. With pg_auto_failover, we avoid the need for complex consensus layers like etcd or Consul while still achieving fast, reliable failover and recovery.

Database layer

Two PostgreSQL nodes are deployed in a primary/secondary configuration. These nodes are registered with a dedicated pg_auto_failover monitor, which continuously checks node health and replication status. In the event of a failure, the monitor promotes the secondary to primary with no manual intervention.

Each node is securely configured using scram-sha-256 authentication and self-signed / or owned SSL certificates to ensure encrypted communication within the cluster.

Monitor node (Arbiter)

The monitor node is a lightweight PostgreSQL instance that runs the pgautofailover extension. It holds state information about all participating nodes and acts as the arbiter during failover events. It requires only one node, reducing complexity compared to consensus-based DCS (Distributed Configuration Store) systems like etcd or ZooKeeper.

Load balancing layer

Two HAProxy nodes route all client (Zabbix) connections to the current PostgreSQL primary. A lightweight HTTP service on each DB node reports its current role (primary or not) and allows HAProxy to determine which node is writable. These proxies are kept highly available using Keepalived, which manages a shared Virtual IP (VIP) across both proxy servers.

This way, applications like Zabbix always connect to a stable endpoint, even during failover events.

Backup layer

Backups are handled using PgBackRest, deployed on a dedicated backup server. This server connects to both PostgreSQL nodes over SSH and performs the following:

Full and incremental backups
WAL archiving
Point-In-Time Recovery (PITR)

Passwordless SSH and proper pgbackrest.conf mappings are set up to support seamless interaction regardless of which node is currently primary.

Component overview

Component	Role
PostgreSQL	Relational backend storing all Zabbix metrics, alerts, events
pg_auto_failover	Ensures continuous availability by promoting replicas automatically
Monitor Node	Decides failover based on health checks and cluster state
HAProxy	Routes client traffic to the current primary
Keepalived	Provides VIP failover between HAProxy nodes
PgBackRest	Performs PITR-capable backups from any node
Zabbix Server	Connects to PostgreSQL via VIP to ensure continuity

Topology at a glance

Design

Unlike Patroni, which requires a distributed configuration store like etcd, pg_auto_failover uses a dedicated monitor node that simplifies orchestration. This setup reduces the operational burden while still delivering robust failover, automatic reconfiguration, and synchronization safeguards, including:

Synchronous_standby_names to enforce replication integrity
Service integration with systemd for reliable restarts
Failover detection with minimal latency

This design also ensures SSL-enabled encrypted communication, self-healing role changes, and full observability using Zabbix itself, which can be configured to monitor the PostgreSQL cluster through exposed health endpoints.

Real-world considerations

Upgrade Planning: The pg_auto_failover version in RPM repos may lag behind the latest upstream features like set_monitor_setting. Pin the package version if consistency is required.
Network Security: Only HAProxy nodes are allowed to query the internal role-check API on the DB nodes using custom firewall rules.
Cluster Hygiene: Always clean up config folders (~postgres/.config/pg_autoctl/…) if a node is misconfigured or needs to rejoin.
SELinux: Configure SELinux, use semanage and audit2allow to fix custom ports (e.g., 9877 for health checks).
Hybrid Logging: Setup PostgreSQL to log to both journald and traditional log files via stderr + logging_collector.

Conclusion

This architecture strikes a balance between simplicity and resilience. While Patroni is great for large-scale, multi-region setups requiring distributed consensus, pg_auto_failover offers a lighter-weight solution that covers most enterprise needs without complex dependencies.

By layering the following…

PostgreSQL 17
Pg_auto_failover with a single monitor
HAProxy + Keepalived for VIP failover
PgBackRest for backups

…you can then confidently run Zabbix in a highly available and secure fashion with minimal operational overhead.

If you’re considering implementing this setup or migrating from a single-node database backend, reach out to Opensource ICT Solutions, a Zabbix Premium Partner with global presence in the USA, the UK, the Netherlands, and Belgium. We can help you architect, deploy, and monitor Zabbix environments that scale with your needs.

The post Running Zabbix with PostgreSQL and PG Auto Failover appeared first on Zabbix Blog.

Migrating Nagios to Zabbix: Lessons Learned

2025-08-05 Nathan Liefting

Post Syndicated from Nathan Liefting original https://blog.zabbix.com/migrating-nagios-to-zabbix-lessons-learned/30917/

Recently, a new customer of ours at Opensource ICT Solutions asked whether we could migrate their Nagios instance to Zabbix. Because Nagios and Zabbix are very different in their storage methods, we told them that we would have to investigate and see if we could come up with a viable solution. It wasn’t long until we found a way to do it and started building some script to get it done.

Table of Contents

The customer’s wishes

No loss of any Nagios configuration data
Historic performance data migrated to Zabbix
Existing problems migrated from Nagios
Nagios XI to be disabled entirely, as the license is expiring

The customer was clear in their wishes – we needed to turn off Nagios, but without losing historic data. As such, they wanted all their old data visible in Zabbix instead of having Nagios running somewhere as a backup. This meant that a script had to be built to get that Nagios data out and into Zabbix.

The configuration data

The good part here is that it starts simple. When we dive into the Nagios configuration data, we clearly see that Nagios has hosts just like Zabbix. They just have a slightly different build than our usual Zabbix hosts. For example, we can see three different names for a host in Nagios:

Host Name
Alias = Host name
Display Name = Visible name

That immediately gives us a good way to hook up Nagios names to Zabbix host and visible names.

When we then take a look at the checks and how they are executed in Nagios, we also see similarities with Zabbix. In the end, both of them are monitoring solutions, of course. However, Nagios works more in a command execution kind of way, which is good for our migration. We can take this command and find an equivalent item in Zabbix. For example the check_icmp command can easily be translated into a simple check in Zabbix icmpping, icmppingloss, and icmppingsec.

For the check_tcp command we can do a similar translation. Making sure we use the simple check net.tcp.service whenever this command is executed on a Nagios host.

Because of the big differences between Nagios and Zabbix, this does mean we need to make some manual translations between the Nagios commands and Zabbix items. Depending on your Nagios instance, this could be a big task. Luckily for us, this was a smaller instance with only ICMP and TCP port checks.

The history (i.e. performance) data

Now that we know how to start creating our hosts and items, we need to understand how Nagios is storing its data. Zabbix has a big centralized MariaDB or PostgreSQL database, which makes it easy to parse through and work with our data. Unfortunately, Nagios instances use a different technique. Nagios stores data in .rrd (Round Robin Database) files and with it a .xml file to interpret the RRD file. The RRD files are not centralized like a Zabbix database, but they are more manageable in terms of storage size. We can see an RRD file per type of check in Nagios, which means we will have to grab the data from that file while understanding what it is going to belong to in Zabbix.

To see the data in the RRD file, we can use a special command line tool.

rrdtool /usr/local/nagios/share/perfdata/BeNeLux-Host-Name/Availability.rrd LAST --start -30d --end now | grep -v "nan"

Now we can clearly see that this specific RRD file above contains 8 columns, 7 with a performance value. The first column contains the timestamp in Unixtime, which is great because it will be perfect for storing in the Zabbix database. The other 7 columns in this file are different though, because we do not know what the value in the column belongs to. This is where the .xml file comes into play. The XML file belongs with the RRD file and contains details on what is included in the RRD file.

In this XML file we will find all of the required host information, which is great for creating the host in Zabbix. It also contains the check information, so we can also use this file to create the items in Zabbix. The biggest thing we will have to keep in mind is to make sure that the XML and RRD file match up in terms of number of RRD entries and columns. Column 1 in the RRD file will match with the first entry in our XML.

Let’s create a script

With the host, item and history data identified, we can start to create a script. In our case we decided to create a Python import tool. As Zabbix comes with some limitations in terms of which hostnames we can use (which are different from the limitations in Nagios), we need to sanitize our hostnames slightly.

Then all we need to do is parse through all the XML files and create new hosts in Zabbix through the Zabbix API.

It will be a very similar process for our items, as we parse through our XML file and create all of the required items in Zabbix through the API.

We can even create the triggers straight from the XML file by parsing through the different severities already set up in Nagios.

Once everything is created in Zabbix, the Python script can now start using RRDTool to parse through the RRD file, making sure to keep the XML file structure in mind when parsing through the columns.

This script can now create the hosts, the items, the triggers, and then import all of the data. We can see the hosts being created and data being imported.

The beauty of importing history data into Zabbix while the triggers are already created is then also seen below.

All of the triggers will trigger and be resolved based on the data imported, meaning that we can create problems with historic data. This means that not only do we have our historic data, but also all the problems with the correct duration as they are now discovered from the actual imported data.

To make this possible we can use the Zabbix sender tool. It has an option to include the timestamp upon every historic value imported.

Our Python script grabs the values from the RRD file and then converts them into a new _HOST_.sender file. This file will be sent to the Zabbix server using the Zabbix sender tool.

Looking at the file, we can see it contains only the name of the host, the unixtime stamp, and the actual value to send.

All we need to do is make our script send this file to the correct item in Zabbix.

Manual template and item creation

The last step will be our cleanup. We decided that we would start dirty with a one-on-one data import from Nagios. This means hostnames, item names, and trigger names are imported straight from Nagios. No templates will be created in Zabbix by the tool either, skipping the Zabbix best practice to use templates for all hosts.

We did this to make the initial import easier and not go overboard with scripting. It’s easier to have a messy Zabbix to clean up than to script everything perfectly in Python. Time is valuable.

What we did afterwards is create all the templates manually to take over the items as is from the hosts. For example, we can translate the ICMP ping and TCP stuff easily into a template.

After doing so, we do end up with some bad looking templates, but we can now start cleaning up.

We can also start creating normal trigger names and clean up…

…while changing our dynamic port names for something more expected as well.

And that’s it!

The post Migrating Nagios to Zabbix: Lessons Learned appeared first on Zabbix Blog.

How to Install Zabbix on Windows with a Linux Subsystem

2025-07-08 Alexander Petrov-Gavrilov

Post Syndicated from Alexander Petrov-Gavrilov original https://blog.zabbix.com/how-to-install-zabbix-on-windows-with-a-linux-subsystem/30311/

It’s a very well known fact that Zabbix can only be installed on Linux. But what if you are in a Windows environment and getting a Linux machine is not so simple or even possible? This can obstruct the implementation of Zabbix, or at least significantly delay it. Not only that, building a POC outside of the future environment makes data procurement a lot more complicated. Is there a way to work around this and get Zabbix as close to Windows as we possibly can?

Table of Contents

WSL

WSL/WSL 2 is a fast and easy solution for installing and using Zabbix in a smaller Windows-dominant environment, be that a POC or a small company office. WSL 2 runs a real Linux kernel in a lightweight VM while being optimized for Windows. This means a faster start, lower resource consumption, and the ability to share files with Windows directly, meaning you can use Windows File explorer to find and manage the VM files.

WSL 2 also allows you to use Linux CLI while working with Windows (i.e. running vim from a Windows terminal and editing Windows files directly). At this point, you may be asking yourself, “Why not Hyper-V and VirtualBox?” Those are definitely options too, but they are quite heavy on system resources. In addition, boot times are a bit longer and sharing files between a host and a guest OS is clunkier.

Maybe Docker Desktop then? It’s an absolutely valid option, but that would require a bit of Docker knowledge and you would still be using WSL, technically speaking. So, with that said, WSL is definitely the fastest and most reliable way to sprung a Zabbix instance in a Windows-focused environment.

We will use WSL 2, but as a note WSL 1 is also available. Here are the differences:

WSL 2 is usually the better performer overall, especially for dev environments. It also has better Linux compatibility.
WSL 1 Linux files aren’t isolated, which can make them more accessible. In WSL 2, Linux runs in a virtual disk (ext4), so Linux and Windows files are more separate. Integration is still pretty good, however.
WSL 2 has better Linux compatibility – systemd, iptables, etc.
WSL 1 shared the same IP as Windows, WSL 2 is a VM – some networking required.
With WSL 1 you can see Linux running processes in Task Manager. WSL 2 will have processes isolated.

Installing Zabbix using WSL

Install WSL

Open PowerShell as an Administrator and run:

PS C:\Windows\system32> wsl --install

If you’ve already have WSL 1 installed, update it:

PS C:\Windows\system32> wsl --update

You can also set WSL 2 as default:

PS C:\Windows\system32> wsl --set-default-version 2

Install/Get preferred Linux Distribution using either Microsoft Store (i.e. Ubuntu, Debian, Oracle Linux) or just download directly. I will be using Oracle Linux 9.4.

You can also download the RootFS tarball from the preferred distribution portal, but then the process will be a bit different. Create a folder using PowerShell:

PS C:\Windows\system32> mkdir C:\WSL\OracleLinux9

Copy the .tar.xz file to this folder, then run:

PS C:\Windows\system32> wsl --import OracleLinux9 C:\WSL\OracleLinux9 .\oraclelinux9-rootfs.tar.xz --version 2

After the image is installed or imported, start Oracle Linux using PowerShell:

PS C:\Windows\system32> oraclelinux94

When installation is finished, there is a prompt to create a default UNIX user account and password for the said user, as the username does not need to match your Windows username. I’ll set it to “zabbix” of course, but you can set it to any other.

PS C:\Windows\system32> Enter new UNIX username: 
PS C:\Windows\system32> zabbix
PS C:\Windows\system32> New password: <your-password>
PS C:\Windows\system32> passwd: all authentication tokens updated successfully.
PS C:\Windows\system32> Installation successful!

Now OracleLinux is ready for use!

Prepare the system

You will be immediately logged in to the new environment. If logged out, to log in again just execute in PowerShel:

PS C:\Windows\system32> oraclelinux94

Being logged in, first double check that your selected OS is indeed installed by executing in the PowerShell, which will now serve as your VM CLI access point:

[zabbix@PC-NAME ~]$ cat /etc/os-release
NAME="Oracle Linux Server"
VERSION="9.4"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Oracle Linux Server 9.4"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:9:4:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL=https://github.com/oracle/oracle-linux

Confirmation received, make sure all OS updates are installed:

[zabbix@PC-NAME ~]$ sudo dnf update -y

When the update process is finished, you will need to decide whether you would like to use systemd or not (this may increase booting time). I will enable systemd. To do this, edit the wsl.conf on the Linux subsystem:

vi /etc/wsl.conf

Add to the newly created file:

[boot]
systemd=true

Reboot the images (this command will reboot all of them):

PS C:\Windows\system32> wsl.exe --shutdown

Start back your Linux distribution:

PS C:\Windows\system32> oraclelinux94

Install Zabbix database

We will need to prepare the database engine. Again, any preferred database engine can be used, in this case I install and configure MariaDB:

[zabbix@PC-NAME ~]$ sudo dnf install -y mariadb-server mariadb
[zabbix@PC-NAME ~]$ sudo systemctl enable --now mariadb

Confirm MariaDB is running:

[zabbix@PC-NAME ~]$ Systemctl status mariadb

mariadb.service - MariaDB 10.5 database server
     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: disabled)
     Active: active (running) since Tue 2025-04-29 12:39:54 EEST; 3min 55s ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
   Main PID: 235 (mariadbd)
     Status: "Taking your SQL requests now..."
      Tasks: 9 (limit: 26213)
     Memory: 109.6M
     CGroup: /system.slice/mariadb.service
             └─235 /usr/libexec/mariadbd --basedir=/usr

After confirmation, secure it a bit by creating a root password and selecting the options in bold:

[zabbix@PC-NAME ~]$ sudo mysql_secure_installation 

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password or using the unix_socket ensures that nobody
can log into the MariaDB root user without the proper authorisation.

You already have your root account protected, so you can safely answer 'n'.

Switch to unix_socket authentication [Y/n] n
 ... skipping.

You already have your root account protected, so you can safely answer 'n'.

Change the root password? [Y/n] Y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!

Remove anonymous users? [Y/n] Y
 ... Success!

Disallow root login remotely? [Y/n] Y
 ... Success!

Remove test database and access to it? [Y/n] Y


Reload privilege tables now? [Y/n] Y

 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

Now to create the Zabbix database. Log in to MariaDB:

[zabbix@PC-NAME ~]$ sudo mysql -u root -p 
[zabbix@PC-NAME ~]$ Enter password: <enter your password, won’t be visible>

Follow the steps from the Zabbix installation page:

MariaDB [(none)]> create database zabbix character set utf8mb4 collate utf8mb4_bin;
MariaDB [(none)]> create user zabbix@localhost identified by '<custom-password>';
MariaDB [(none)]> grant all privileges on zabbix.* to zabbix@localhost;
MariaDB [(none)]> set global log_bin_trust_function_creators = 1;

MariaDB [(none)]> quit;

Installing Zabbix

Install the Zabbix repository:

[zabbix@PC-NAME ~]$ sudo dnf install https://repo.zabbix.com/zabbix/7.0/centos/9/x86_64/zabbix-release-latest-7.0.el9.noarch.rpm
[zabbix@PC-NAME ~]$ dnf clean all

Proceed to install the Zabbix server, frontend, and agent:

[zabbix@PC-NAME ~]$ sudo dnf -y install zabbix-server-mysql zabbix-web-mysql zabbix-apache-conf zabbix-sql-scripts zabbix-selinux-policy zabbix-agent
...
[zabbix@PC-NAME ~]$zabbix-agent-7.0.12-release1.el9.x86_64 zabbix-apache-conf-7.0.12-release1.el9.noarch zabbix-selinux-policy-7.0.12-release1.el9.x86_64  zabbix-server-mysql-7.0.12-release1.el9.x86_64 zabbix-sql-scripts-7.0.12-release1.el9.noarch
 zabbix-web-7.0.12-release1.el9.noarch zabbix-web-deps-7.0.12-release1.el9.noarch zabbix-web-mysql-7.0.12-release1.el9.noarch

Complete!

Now import the initial database schema:

[zabbix@PC-NAME ~]$ zcat /usr/share/zabbix-sql-scripts/mysql/server.sql.gz | mysql -u zabbix -p zabbix
Enter password: <enter your DB user password and wait until you will see the next line appear>
[root@ZBX-5CD3221K14 zabbix]#

Disable the log_bin_trust_function_creators option after import has finished:

# mysql -uroot -p
password
MariaDB [(none)]>  set global log_bin_trust_function_creators = 0;
MariaDB [(none)]>  quit;

Add your Zabbix user database password to the Zabbix server configuration file:

[zabbix@PC-NAME ~]$ vi /etc/zabbix/zabbix_server.conf

### Option: DBPassword
#       Database password.
#       Comment this line if no password is used.
#
# Mandatory: no
# Default:
DBPassword=<your-DB-user-password>

Start the Zabbix server and frontend and add them to autorun:

[zabbix@PC-NAME ~]$  systemctl restart zabbix-server zabbix-agent httpd php-fpm
[zabbix@PC-NAME ~]$  systemctl enable zabbix-server zabbix-agent httpd php-fpm
Created symlink /etc/systemd/system/multi-user.target.wants/zabbix-server.service → /usr/lib/systemd/system/zabbix-server.service.
Created symlink /etc/systemd/system/multi-user.target.wants/zabbix-agent.service → /usr/lib/systemd/system/zabbix-agent.service.
Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service → /usr/lib/systemd/system/httpd.service.
Created symlink /etc/systemd/system/multi-user.target.wants/php-fpm.service → /usr/lib/systemd/system/php-fpm.service.

Installation of the backend is now finished, but we still need the frontend.

Exposing and installing the Zabbix frontend for WSL

Since WSL2 does not expose services to localhost by default, you need to determine the WSL IP:

[zabbix@PC-NAME ~]$ ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:47:32:c6 brd ff:ff:ff:ff:ff:ff
    inet 172.29.128.155/20 brd 172.29.143.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::215:5dff:fe47:32c6/64 scope link
       valid_lft forever preferred_lft forever

Look for an IP like 172.x.x.x, then using your browser go to:

http://<WSL_IP>/zabbix

In this example, that would be:

http://172.29.128.155/zabbix

You can also port forward WSL to localhost with netsh in PowerShell:

PS C:\Windows\system32> netsh interface portproxy add v4tov4 listenport=8080 listenaddress=127.0.0.1 connectport=80 connectaddress=<WSL_IP>

Then you will be able to access Zabbix from http://localhost:8080/zabbix. Now, just finish the standard frontend setup and Zabbix is ready to use!

WSL advantages

Some extra advantages you get with this approach include clearer resource usage visibility:

Direct access to the Linux subsystem files through File explorer with your favorite Windows tools:

As you can see, docker is here as well. System and configuration files are also visible and editable:

Now you can proceed with building your Zabbix or Zabbix POC, (almost) without needing to leave your regular Windows environment!

The post How to Install Zabbix on Windows with a Linux Subsystem appeared first on Zabbix Blog.

Podman Container Monitoring with Prometheus Exporter, part 2

2025-06-12 Janis Eidaks

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/podman-container-monitoring-with-prometheus-exporter-part-2/30538/

In the first part of this post, we explored how to get data with HTTP agent from the Prometheus Podman exporter and use the same item data for the Podman pods Discovery rule as well as item and trigger prototypes. In part 2 of the same series, we’ll learn how to discover and monitor Podman containers.

Table of Contents

Creating a template discovery rule

I will create another discovery rule for container discovery. This discovery rule is also based on the same item [Podman info] in the template – Podman containers by HTTP and Prometheus (you can check part one of this series to find out how to configure it). The parameters of the discovery rule are shown below. This discovery rule will allow us to discover the pod name and ID.

Template: Podman containers by HTTP and Prometheus

▲ Discovery rule
  ▪ Name:                   Container discovery
  ▪ Type                    Dependent item
  ▪ Key:                    training.containers.discovery
  ▪ Master item             Podman containers by HTTP and Prometheus: Podman info
  ▪ Delete lost resources  After 10d
  ▪ Disable lost resources Immediately
♯ Preprocessing
  ▪ Prometheus to JSON     podman_container_info
♦ LLD Macros
  ▪ {#CONTAINER.ID}        $.labels.id
  ▪ {#CONTAINER.NAME}      $.labels.name

Fig 1. Discovery rule: Container discovery

Fig 2. Discovery rule: Container discovery preprocessing tab

Fig 3. Discovery rule: Container discovery LLD macros tab

Next, different dependent item prototypes are created in this container discovery rule. As the Prometheus Podman exporter provides a lot of different metrics about the containers, I will create multiple such items: state, health, creation date, input/output network traffic information, and so on. So, check out what metrics can be acquired and use what is relevant for you.

You can also add a description of each item prototype. I am interested only in metrics with the discovered container ID macros, and I am not interested in what values are for the other fields, such as pod_id, pod_name, so I use ~”.*”, which matches any value. I will show the item prototype configuration screenshots of one of the item prototypes.

These item prototypes are similar to each other, with some minor differences, such as Prometheus patterns, or in some cases, with a different master item (item prototype as master item).

Fig 4. Discovery rule preprocessing step: Prometheus to JSON with pattern podman_container_info

Fig 5. Discovery rule LLD macros: assigning relevant JSONPATH to LLD macros

Creating a template discovery rule: Item prototypes

After the containers have been discovered, we have to create item prototypes. These prototypes will also be dependent item prototypes and will use the same item as the discovery rule: Podman info. Prometheus Podman exported returns a lot more metrics for the containers than it did for the pods.

You can get container metrics such as container health, state, creation date, disk read/write, memory usage, network usage, and more. In this blog post, I have added most of them, so check what metrics are relevant to your monitoring needs and start monitoring.

Fig 6. Low-level discovery rule and item prototypes based on the same item.

The screenshots of the item prototype is shown below.

Fig 7. Container state item prototype tab

Fig 8. Container state item prototype tag tab

Fig 9. Container state item prototype preprocessing tab

Remember, you can also test these item prototypes in the preprocessing step – just copy the Prometheus exporter data and set the relevant macro to value you want to check.

The configuration parameters of the item prototypes are shown below. There are a lot of metrics you can monitor, but remember to monitor what is relevant and necessary for you.

Template: Podman containers by HTTP and Prometheus; Discovery rule: Container discovery

● Item prototype #1
  ▪ Name: 		Container health: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.health[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (float)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 
♦ Tags (name:value) 	
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:health		
♯ Preprocessing
  ▪ Prometheus pattern 	podman_container_health{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #2
  ▪ Name: 		Container state: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.state[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (float)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		
♦ Tags (name:value)  			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:state		
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_state{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #3
  ▪ Name: 		Created at: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.created[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		unixtime
♦ Tags (name:value) 		
  ▪ Container:{#CONTAINER.NAME}	
  ▪Metric:created		
♯ Preprocessing
  ▪ Prometheus pattern 	podman_container_created_seconds{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #4
  ▪ Name: 		Disk read per second: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.disk.read[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		B
♦ Tags (name:value) 	
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:disk_read		
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_block_output_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value
  ▪ Change per second

● Item prototype #5
  ▪ Name: 		Disk write per second: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.disk.write[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		B
♦ Tags (name:value) 	
  ▪ Container:{#CONTAINER.NAME}	 
  ▪ Metric:disk_write		
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_block_input_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value
  ▪ Change per second

● Item prototype #6
  ▪ Name: 		Exit code: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.exit_code[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (float)
    ▪ Master item	Podman containers by HTTP and Prometheus: Podman info
▪ Units: 			
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:exit_code	
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_exit_code{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #7
  ▪ Name: 		Image tags: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.image.tags[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Character
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 			
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:tag
♯ Preprocessing
▪ Prometheus pattern podman_container_info{id="{#CONTAINER.ID}",image=~".*",name=~".*",pod_id=~".*",pod_name=~".*",ports=~".*"} label image
  ▪ Regular expression	\.*(\/.\w.*)	\1

● Item prototype #8
  ▪ Name: 		Memory usage: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.mem[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		B
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:mem		
♯ Preprocessing
  ▪ Prometheus pattern podman_container_mem_usage_bytes{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #9
  ▪ Name: 		Network input dropped: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.in.drop[{#CONTAINER.NAME}]
  ▪ Type of inf: Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		packets
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_in_drop		
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_net_input_dropped_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #10
  ▪ Name: 		Network input errors: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.in.errors[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_in_err		
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_net_input_errors_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #11
  ▪ Name: 		Network input total: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.in.total[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		B
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_in_tot
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_net_input_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #12
  ▪ Name: 		Network input per second: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.in.change[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (float)
  ▪ Master item		prototype - Network input total: [{#CONTAINER.NAME}] 
  ▪ Units: 		Bps
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_in_change
♯ Preprocessing
  ▪ Change per second

● Item prototype #13
  ▪ Name: 		Network output dropped: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.out.drop[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_out_drop	
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_net_output_dropped_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #14
  ▪ Name: 		Network output errors: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.out.errors[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_out_err	
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_net_output_errors_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #15
  ▪ Name: 		Network output total: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.out.total[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		B
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_out_tot	
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_net_output_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #16
  ▪ Name: 		Network output per second: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.out.change[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (float)
  ▪ Master item		prototype - Network output total: [{#CONTAINER.NAME}]
  ▪ Units: 		Bps
♦ Tags 			 
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_out_change
♯ Preprocessing
  ▪ Name			Change per second

● Item prototype #17
  ▪ Name: 		Rootfs size: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.rootfs.size[{#CONTAINER.NAME}]
  ▪ Type of inf: Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		B
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:rootfs
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_rootfs_size_bytes{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #18
  ▪ Name: 		Total system CPU time: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.cpu.time
  ▪ Type of inf: 	Numeric (float)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		s
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:sys_time
♯ Preprocessing
  ▪ Prometheus pattern: podman_container_cpu_system_seconds_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

Creating a template discovery rule: Trigger prototype

I have created a user macro {$CONTAINER.RUNNING.STATE} on the template with a value of 2, which corresponds to the containers running state. After that, create a trigger prototype to check if the container is in different state other than running.

Template: Podman containers by HTTP and Prometheus; Discovery rule: Container discovery

◘ Trigger prototypes
  ▪ Name:               Container [{#CONTAINER.NAME}] state has changed from running
  ▪ Severity:           Warning
  ▪ Expression:         last(/Podman containers by HTTP and Prometheus/container.state[{#CONTAINER.NAME}])<>{$CONTAINER.RUNNING.STATE}
  ▪ PROBLEM event generation mode: Single
  ▪ OK event closes: All problems

So, once all of this is done, and some container status changes from running and or pod status also changes from running, you will get a problem event.

Fig 10. Generated problem events when the podman pod and container change states.

Technically, I could also create a trigger for container health; however, as all of the received container values for me are -1 (meaning unknown) it makes little sense to make a trigger that will fire right away. You can also add additional item/trigger prototypes in the template. If everything is set up as expected, you should see something like the screenshot below after the LLD rule execution.

Fig 11. Example of the mysql-server container and zabbix pod item values.

Summary

Now, you can monitor both Podman pods and containers using both blog posts of this series. We used the same template item for both the container LLD and item prototypes from the first part of this post.

The post Podman Container Monitoring with Prometheus Exporter, part 2 appeared first on Zabbix Blog.

Podman Container Monitoring with Prometheus Exporter, part 1

2025-06-10 Janis Eidaks

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/podman-container-monitoring-with-prometheus-exporter-part-1/30513/

In part one of this blog post, I will show you how to monitor Podman pods using HTTP agent item to retrieve data from the Prometheus Podman exporter. Let’s get started!

Table of Contents

Installing and checking Prometheus Podman exporter

First, you will need to install and enable the Prometheus Podman exporter (my OS is CentOS Stream release 9). Then, check that the service is active and running.

# dnf install -y prometheus-podman-exporter

# systemctl enable prometheus-podman-exporter –now

# systemctl status prometheus-podman-exporter

You can check that you are getting the data from the exporter with either the curl command from the machine/VM where the Prometheus podman exporter is installed and started:

# curl http://localhost:9882/metrics

Fig 1. Output of Prometheus podman exporter in CLI

Or through the browser (replace abc with the machine’s IP/DNS ): abc:9882/metrics.

Fig 2. Output of Prometheus Podman exporter in browser

A line starting with # is a comment and contains an explanation regarding the metric; in this case, podman_container_block_input_total will return data in bytes. In Figure 2, after the comments, you can see several podman_container_block_input_total metrics, one for each container, with different container IDs, pod IDs, and pod names listed in each metric. The metric’s value is displayed on the right side after curly brackets.

Creating a template and template items

Next, I will create a template Podman containers by HTTP and Prometheus where I will put all of the entities (everything will be created on the template). In the template, I will create an item Podman info, which will gather all of the necessary data at defined intervals. This approach will be convenient from a data collection standpoint as the same item data will be used for LLD and item prototypes. During testing, you can set “History” to store data for some time, and when everything is working as expected, then set “History” not to keep any data. This item will be used for the Low-Level Discovery rule and the item prototype.

The item Podman info parameters are as follows:

Template: Podman containers by HTTP and Prometheus

○ Item
  ▪ Name:         Podman info
  ▪ Type          HTTP agent
  ▪ Key:          podman.info
  ▪ Type of inf   text
  ▪ URL           http://{HOST.CONN}:9882/metrics
  ▪ Request type  GET
  ▪ Update int.   5m
  ▪ Req status c. 200
  ▪ History       Do not store
◊ Tags
  ▪ Podman:raw

At this moment, this item will contain just raw data, without any preprocessing steps applied. The IP address will be taken from any host interface added to the host. You will get an error message if the host has no interface.

Fig 4. Error on the host with the linked template without any interface

If you do not want to add an interface to the host, you can define a user macro on the template level and use that user macro in the items URL. After adding the template to the host, just modify the user macro value on the host to correct IP/DNS name.

Fig 6. Template item for data gathering with user macro instead of built in macro from host interface

I can also create an item to determine the number of containers created. I can count specific Prometheus pattern occurrences in the master item to determine this. For this, I will use the podman_container_state parameter. Likewise, I could use different parameters, such as podman_container_info, and count the occurrences of such a pattern. The parameters of the item container count:

Template: Podman containers by HTTP and Prometheus

○ Item
  ▪ Name:         Container count
  ▪ Type          Dependent item
  ▪ Key:          container.count
  ▪ Type of inf   Numeric (unsigned)
  ▪ Master item   Podman containers by HTTP and Prometheus: Podman info
◊ Tags
  ▪ Containers:total
♯ Preprocessing
  ▪ Prometheus pattern     podman_container_state     count

Fig 7. Template item preprocessing step for counting the total number of containers

Creating a Discovery rule in template

Next, the LLD rule will be created to discover Podman pods. It will be a dependent LLD rule based on a Podman info item with a preprocessing step to convert the Prometheus pattern data to JSON format. The caveat is that the LLD discovery will be executed as frequently as the data is received for the item. If there are a lot of hosts with such a template, there will be a lot of LLD processes executed, which can put a strain on your Zabbix instance.

To rectify this issue, I will add a preprocessing step: discard unchanged with heartbeat (as there are no dynamic parameters in the extracted pattern, otherwise we would need to filter out dynamically changing information). For LLD discovery, the recommended interval is around 1h. Additionally, LLD macros will be created from selected JSNOPath variables. The parameters of the LLD rule are shown below.

Template: Podman containers by HTTP and Prometheus

▲ Discovery rule
  ▪ Name:                   POD discovery
  ▪ Type                    Dependent item
  ▪ Key:                    training.pod.discovery
  ▪ Master item             Podman containers by HTTP and Prometheus: Podman info
  ▪ Delete lost resources  After 10d
  ▪ Disable lost resources Immediately
♯ Preprocessing
  ▪ Prometheus to JSON     podman_pod_info
  ▪ Discard unchanged with heartbeat 1h
♦ LLD Macros
  ▪ {#POD.ID}              $.labels.id
  ▪ {#POD.NAME}            $.labels.name

Fig 9. Discovery rule: Pod discovery preprocessing tab

Fig 10. Discovery rule: Pod discovery LLD macros tab

The block diagram below will show how the data is transformed. First, a preprocessing step is applied to the data to convert the Prometheus pattern to JSON format, as all data for LLD must be supplied in JSON format.

In the example below, the matching queried pattern is returned in JSON format after this preprocessing step.

Fig 11. Discovery rule preprocessing step: Prometheus to JSON with pattern podman_pod_info

After the preprocessing step, we can assign specific JSONPATH values to LLD macros.

Fig 12. Discovery rule LLD macros: assigning relevant JSONPATH to LLD macros

Creating a template Discovery rule: item prototypes

Now that we have discovered the macros we are interested in, the discovered macros can be used for further prototype (ITEM/HOST/TRIGGER) creation. In this example, I am using the same master item for LLD discovery and the dependent item prototypes, because it is convenient for me, and all the information is available in one item. But usually, there are scenarios where you have to use one item’s data for discovery and the data of another item for populating the prototype values.

In this case, I am interested in the pod ID, when the pod was created, the number of containers in the pod, and the state of the pod. Therefore, I will create the item prototypes and use the LLD macro in the name, key, and preprocessing step. Zabbix will cycle through the discovered LLD macro values and create the items based on the prototype by replacing the LLD macro with discovered values. Although you can set matching item prototype names (which will be confusing), you still have to use the LLD macro in the item key so that different item keys are generated – otherwise, you will get an error regarding duplicate keys. The item prototype parameters are given below.

Fig 13. Low-level discovery rule and item prototypes based on the same item.

Template: Podman containers by HTTP and Prometheus; Discovery rule: POD discovery

○ Item prototype #1
  ▪ Name:         POD ID: [{#POD.NAME}]
  ▪ Type          Dependent item
  ▪ Key:          pod.id[{#POD.NAME}]
  ▪ Type of inf   Character
  ▪ Master item   Podman containers by HTTP and Prometheus: Podman info
♦ Tags
  ▪ Metric:ID
  ▪ Pod:{#CONTAINER.NAME}
♯ Preprocessing
  ▪ Prometheus pattern     podman_pod_containers{id="{#POD.ID}"}          label    id

○ Item prototype #2
  ▪ Name:         POD state: {#POD.NAME}
  ▪ Type          Dependent item
  ▪ Key:          pod.state[{#POD.NAME}]
  ▪ Type of inf   Numeric (float)
  ▪ Master item   Podman containers by HTTP and Prometheus: Podman info
  ▪ Value mapping POD state
♦ Tags
  ▪ Metric:state
  ▪ Pod:{#CONTAINER.NAME}
♯ Preprocessing
  ▪ Prometheus pattern     podman_pod_state{id="{#POD.ID}"}      value

○ Item prototype #3
  ▪ Name:         POD created at: [{#POD.NAME}]
  ▪ Type          Dependent item
  ▪ Key:          pod.created[{#POD.NAME}]
  ▪ Type of inf   Numeric (unsigned)
  ▪ Units         unixtime
  ▪ Master item   Podman containers by HTTP and Prometheus: Podman info
♦ Tags
  ▪ Metric:created
  ▪ Pod:{#CONTAINER.NAME}
♯ Preprocessing
  ▪ Prometheus pattern     podman_pod_created_seconds{id="{#POD.ID}"}     value

○ Item prototype #4
  ▪ Name:         POD container count: [{#POD.NAME}]
  ▪ Type          Dependent item
  ▪ Key:          pod.count[{#POD.ID}]
  ▪ Type of inf   Numeric (unsigned)
  ▪ Master item   Podman containers by HTTP and Prometheus: Podman info
♦ Tags
  ▪ Metric:count
  ▪ Pod:{#CONTAINER.NAME}

On the template, I have also created a value map for deciphering the numerical pod state codes to text strings for better clarity.

Fig 14. Value mapping for the POD state item

Here are some screenshots of the POD state item prototype, shown below.

Fig 15. POD state item prototype: item prototype tab

Fig 16. POD state item prototype: tag tab

Fig 17. POD state item prototype: preprocessing tab

Creating a template Discovery rule: trigger prototype

We can also create a trigger prototype to generate an alert if there is something wrong with the pod. I have created a user macro {$POD.RUNNING.STATE} on the template with a value of 4, which corresponds to the running state.

Template: Podman containers by HTTP and Prometheus; Discovery rule: POD discovery

◘ Trigger prototypes:
  ▪ Name:               POD [{#POD.NAME}] state has changed from running
  ▪ Severity:           Warning
  ▪ Expression: last(/Podman containers by HTTP and Prometheus/pod.state[{#POD.NAME}])<>{$POD.RUNNING.STATE}
  ▪ PROBLEM event generation mode: Single
  ▪ OK event closes: All problems

Fig 18. Trigger prototype based on POD state item value

Once you link the template to the host and execute the LLD rule, you should start seeing the Podman pods ( if you have them), similar to the screenshot below.

Fig 19. Latest data for the host with the linked template

Summary

This blog post shows how to get data with HTTP agent from Prometheus Podman exporter and use the same item data for the Discovery rule as well as item and trigger prototypes. Check out part 2 of this series to find out how to discover and monitor Podman containers.

The post Podman Container Monitoring with Prometheus Exporter, part 1 appeared first on Zabbix Blog.

Database Monitoring using Zabbix agent 2 – Part 1, SQL

2025-06-03 Alexander Petrov-Gavrilov

Post Syndicated from Alexander Petrov-Gavrilov original https://blog.zabbix.com/database-monitoring-using-zabbix-agent-2-part-1-sql/30381/

If you find yourself needing additional flexibility when it comes to database monitoring, Zabbix agent 2 may be exactly what you need. Keep reading to see which features make it ideal for database monitoring and find out how to best use them for your own purposes.

Table of Contents

What is a database?

If you’ve been using Zabbix for a while, you know that a database is an organized collection of data that is stored and accessed electronically.
That data can be historical, configuration, business, social media-related, etc. A database, or rather a database management system (DBMS) allows you to store, manage, and retrieve information efficiently.

Types of DBMS

We can separate DBMS into multiple types. Depending on how data is stored, retrieved, managed, there can be quite a few, but we will try to limit ourselves to the most common four:

Relational databases (or RDBMS) see tables and SQL.
- MySQL
- MariaDB
- PostgreSQL
- Oracle

NoSQL databases store data in formats like JSON, key-value pairs, or graphs.
- MongoDB
- Redis
- InfluxDB
- ElasticSearch

Cloud databases use cloud platforms for scalability.
- Amazon RDS
- Azure SQL

Time-series databases (or TSDB databases) are optimized for time-stamped data.
- TimescaleDB
- InfluxDB

But what unites all those database engines? They can all be monitored by Zabbix!

Database monitoring

Database monitoring is important for a variety of reasons, the most common of which are to get a precise overview of database and application performance. Since databases can be a vital part of multiple departments and applications, poor performance may impact an entire company and its users, leading to unsatisfactory results on all sides.

To avoid such situations, the set of metrics we should monitor for database engines can include:

Database environment metrics
- CPU performance
- Memory usage
- Drive capacity
- Disk latency

Database performance metrics
- Query performance
- Transaction/operations/indexing
- Connections

Application and/or business related data
- Amount of users
- Transactions
- Inventory
- Configuration

Why Zabbix agent 2?

Zabbix Agent 2 includes multiple features that enhance its flexibility:

Task queue management with respect to both schedule and task concurrency.
Concurrent active checks with threads.
Multiple agent 2 unique metrics
Easier to extend using GO plugins.

Plugins in Zabbix Agent 2 are written in the Go programming language and provide a flexible, native way to extend the agent’s functionality. These plugins communicate directly with databases using their native APIs or libraries, which allows for correct and efficient performance monitoring.

But agent2 provides even more flexibility when focusing on database monitoring, allowing us to:

Limit query execution
Control the session time
Configure encryption between Zabbix agent and database
Control cache mode

All database data is collected using the best approach for the monitored database.

MySQL, monitoring relies on the Go-MySQL-Driver
PostgreSQL integration is managed through the pgx driver

The list goes on for supported database engines:

MySQL / MariaDB
PostgreSQL
ORACLE
MSSQL
MongoDB
Redis
Memcached

Monitoring SQL databases

Database environment

In this part we will focus on how to monitor and retrieve data from SQL databases and SQL database-related parameters. Monitoring SQL database environment metrics with Zabbix agent 2 is as straightforward as monitoring any virtual or physical machine with an OS. All we need to do is add the repo:

# dnf install https://repo.zabbix.com/zabbix/7.0/centos/9/x86_64/zabbix-release-latest-7.0.el9.noarch.rpm

Install the agent:

# dnf install zabbix-agent2

Then, make sure that connections from Zabbix server to Zabbix agent 2 are allowed using Server parameter:

### Option: Server
#       List of comma delimited IP addresses, optionally in CIDR notation, or DNS names of Zabbix servers and Zabbix proxies.
#       Incoming connections will be accepted only from the hosts listed here....
# Mandatory: no
# Default:
# Server=
Server=127.0.0.1,server-dns.example.com

Finally, link one of the many templates available out of the box:

SQL database performance metrics

What about the actual DB performance metrics? There are plenty of approaches we can take using Zabbix agent 2.

Out-of-the-box templates are available for multiple databases that can be monitored by Zabbix agent 2:

Each of the templates uses a database native way to get precise performance data, such as SHOW GLOBAL STATUS for MySQL or dbStats for MongoDB. Also, template provides instructions on how to prepare the database for monitoring. Let’s take MySQL/MariaDB for example:

Create a MySQL user for monitoring (<password> at your discretion) and give this user enough permissions for monitoring:

mysql> CREATE USER 'zbx_monitor'@'%' IDENTIFIED BY '<password>';
mysql> GRANT REPLICATION CLIENT,PROCESS,SHOW DATABASES,SHOW VIEW ON *.* TO 'zbx_monitor'@'%';

In order to collect replication metrics, MariaDB Enterprise Server 10.5.8-5 and above and MariaDB Community Server 10.5.9 and above require the SLAVE MONITOR privilege to be set for the monitoring user. The command then looks like this:

mysql> GRANT REPLICATION CLIENT,PROCESS,SHOW DATABASES,SHOW VIEW,SLAVE MONITOR ON *.* TO 'zbx_monitor'@'%';

Then create a host to represent your MySQL/MariaDB and link the “MySQL by Zabbix agent 2” template:

Configure the Macros on the same host:

And the data will start pouring in!

You can find instruction for other databases here.

SQL database internal data monitoring

A default template will tell us a lot about performance, but what if we also need application data? Something that is stored in the database, i.e.

Number of orders
Logged in users
Host count
List of failed transactions
Amount of media uploaded

Zabbix agent 2 lets users collect custom SQL query results with the help of configuration files and a specific item key:

<dbtype>.custom.query[connString,<user>,<password>,queryName,<args...>]:
• Dbtype – mysql, postgresql, oracle, mssql
• connString - URI or session name;
• user, password - Database login credentials;
• queryName - name of a custom query, matches SQL file name without .sql extension;
• args - one or several comma-separated arguments to pass to a query.

The main idea of this key is to construct efficient queries that can return multiple values. The values returned will be automatically transformed to JSON, which is both easier to preprocess and use for LLD creation.

I will add a simple query to find all hosts and their main interface availability in Zabbix:

SELECT hosts.host,interface.available FROM zabbix.hosts JOIN zabbix.interface ON hosts.hostid=interface.hostid WHERE hosts.status IN (0,1) AND hosts.flags IN (0,4) AND interface.main=1;

First I need to create a directory for custom queries:

# mkdir /etc/zabbix/zabbix_agent2.d/plugins.d/custom_queries

Now I will create an .sql file with a query and paste the mentioned query into the file:

# nano /etc/zabbix/zabbix_agent2.d/plugins.d/custom_queries/interfaces.sql

Now I will edit the MySQL plugin .conf file and set a custom queries path:

# nano /etc/zabbix/zabbix_agent2.d/plugins.d/mysql.conf
### Option: Plugins.Mysql.CustomQueriesPath
#       Full pathname of a directory containing *.sql* files with custom queries.
#
# Mandatory: no
# Default:
# Plugins.Mysql.CustomQueriesPath=
Plugins.Mysql.CustomQueriesPath=/etc/zabbix/zabbix_agent2.d/plugins.d/custom_queries/

Save the changes and restart Zabbix agent 2 to apply them:

# systemctl restart zabbix-agent2

Before adding the item using the web interface, it is always a good idea to test it:

zabbix_agent2 -t mysql.custom.query["tcp://localhost:3306","zbx_monitor","<password>","interfaces"]

The output will is now a easy to work with JSON pattern (beautified here):

[
  {
    "available": "1",
    "host": "Zabbix server"
  },
  {
    "available": "1",
    "host": "Test environment"
  },
  {
    "available": "1",
    "host": "MySQL database"
  },
  {
    "available": "1",
    "host": "MongoDB database"
  },
  {
    "available": "1",
    "host": "PostgreSQL database"
  },
  {
    "available": "1",
    "host": "Customer portal"
  }
]

Now, I’m sure the data is collected and can be used for LLD. I can create a new item on the MySQL database host to collect this data:

Since I know what kind of data will be returned, I can create a dependent Discovery rule on the same host:

The LLD macros tab will help to transform the current JSON to the LLD-suitable JSON, replacing “host” with {#HOST}.

Interface LLD item mecros — Interface LLD item macros

After adding the discovery itself, we can create the dependent item prototype, which will allow us to discover all hosts and their status:

Preprocessing here is a must, and it needs to be flexible enough to extract each individual host interface status:

Interface status item prototype preprocessing

Now after adding the item prototype, we can check the results:

An item cam be further enhanced using value mapping, to specify that 1 means available and 0 means not available.

With this approach, any internal database data can be extracted and monitored. In part 2 we will see how NoSQL databases can be monitored for both performance and internal data using Zabbix agent 2.

If you’d like more information on database monitoring, please don’t hesitate to sign up for our training course in Advanced Zabbix Database Monitoring, which covers multiple approaches to collecting database-related performance metrics and data using Zabbix Agent 2, ODBC, and API requests, as well as optimizing data collection by introducing dependent low-level discovery for minimal performance impact.

The post Database Monitoring using Zabbix agent 2 – Part 1, SQL appeared first on Zabbix Blog.