All posts by Janis Eidaks

Zabbix and the Docker API, Part 3: Control

2026-05-06 Janis Eidaks

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/zabbix-and-the-docker-api-part-3-control/32961/

In this blog post, you will learn how to add a simple container remote control capability to Zabbix in order to start, stop, or restart containers from within the discovered host.

You might be wondering, why spend the effort to create a host for each template? Well, that’s because we define a manual script to control the container from within the Zabbix frontend. That’s neat, right? And why stop there? We can also implement a trigger action that automatically restarts the container if it crashes for any reason.

Zabbix server configuration changes

First, we will require global script execution in your Zabbix server configuration:

# nano /etc/zabbix/zabbix_server.conf
EnableGlobalScripts=1
# systemctl restart zabbix-server

Script configuration in frontend

We can create a script in the section Alerts > Scripts. In the script, fill out the specified parameters shown below – the scope, type, and command. Then specify to which hosts this command will apply, as well as the user group that will be able to execute this. This script will take advantage of the user macros and built-in macros to fill the required info in the command to make a correct post request.

● Script
▪ Name: Container action
▪ Scope : Manual host action
▪ Type: Script
▪ Execute on: Zabbix server
▪ Commands: curl -sS -X POST https://{$DOCKER.IP}:{$DOCKER.PORT}/containers{HOST.NAME}/{MANUALINPUT} --cert /etc/zabbix/ssl/certs/client-cert.pem --key /etc/zabbix/ssl/keys/client-key.pem --cacert /etc/zabbix/ssl/ca/ca.pem
▪ Description: Manual action to restart,stop,start container
▪ Host group: Selected: Docker
▪ User group: Zabbix administrators 
▪ Req host perm: Write

● Advanced configuration
▪ Enable user input Check
▪ Input prompt Specify action for container {HOST.NAME}:
▪ Input type: Dropdown 
▪ Dropdown options: restart,stop,start
▪ Enable Confirmation: Check
▪ Confirmation text: Confirm to {MANUALINPUT} container: {HOST.NAME}

Manual host script execution

We can go the Menu section Monitoring > Hosts, select the host, and click on it. In the menu, you will have an additional script available for the Docker hosts group Container action. This manual action is also available in some other frontend sections.

Fig 2. The available scripts for manual execution on the host

Once you click on the Container action, you will have several options available. You can start, restart, or stop the container.

Fig 3. Drop-down menu options for the script

You will have a confirmation window asking if this is the right action you want to perform.

Fig 5. Script output on successful execution.

The status can also be checked in the host’s latest data menu, once the metric is collected (1 minute for the master item). The item Container /zabbix-agent2: Running shows that this container is not running, and another item displays the exit code 0, which means the process stopped normally with no issue whatsoever.

Fig 6. The latest data for the Zabbix agent 2 container

Some items report the status in numerical format, e.g., 0 , 1 , 2, and so on. To make it human-readable, we use value maps, which display the value in a meaningful, human-friendly way. The screenshot below shows the value map for container health status. So, instead of looking at value 3 for container health (which is meaningless for us and will require reading the documentation) we will be shown value healthy (3).

Fig 7. Predefined value mapping on the template

Automating the container crash recovery

What if your container crashes for some reason? Well, you will get a problem event, which you can use to receive notifications about issues with containers. You can also automate the container recovery process. For example, create a trigger action that will restart the container 3 times with an interval of 2 minutes.

If it does not resolve the issue, only then send a message to the admin. There is no reason to repeatedly restart the service until the end of time – if a few attempts did not work, most likely it will require human intervention to solve the issue.

So here are the script parameters for the action operation:

● Script
▪ Name: Restart container
▪ Scope: Action operation
▪ Type: Script
▪ Execute on: Zabbix server
▪ Commands: curl -sS -X POST https://{$DOCKER.IP}:{$DOCKER.PORT}/containers{HOST.NAME}/restart --cert /etc/zabbix/ssl/certs/client-cert.pem --key /etc/zabbix/ssl/keys/client-key.pem --cacert /etc/zabbix/ssl/ca/ca.pem
▪ Description: Restart container
▪ Host group: Selected: Docker

Now we have to define a trigger action in order to make use of this script and send a notification to admin if that fails.

Let’s create a new trigger action:

● Action tab
▪ Name: Automatic container restart
▪ Type of calc: And (A and B)
▪ condition: Host group equals Docker
▪ condition: Event name contains Container has been stopped with error code
▪ Enabled: Check

● Operations tab
▪ Default operation step duration: 2m

Add operation
▪ Operation: Current host: Check
▪ Steps: 1 -3
Add operation
▪ Operation: Send message
▪ Steps: 4 – 4
▪ Custom message: Check
▪ Subject: Automated restart failed to bring container up: {HOST.NAME}
▪ Message: <b>Host: {HOST.NAME}<br>
           <b>Problem started at {EVENT.TIME} on {EVENT.DATE}<br>
           <b>Operational data: {EVENT.OPDATA}<br>
           <b>Original problem ID: {EVENT.ID}<br>

Fig 10. Action operation tab: new Operation step 1-3

Fig 11. Action operation tab: new step 4-4

The action should look like the screenshot below. Save it.

Testing container crash automatic recovery

I will stop the container with the command docker kill zabbix-agent2. The container has been stopped with an exit code different from 0, so when the item receives the data (in my case, after 1 minute) I get a problem event. The trigger action executes the script Container restart immediately, after 2 minutes, and after 4 minutes if the problem event has not been resolved.

Fig 13. Problem event about stopped container exit code 137

This script successfully restarted the container. The container is running again, and the problem event is resolved.

Fig 14. Resolved event with remote script execution

Let’s see if I am quick enough to kill the west proxy container repeatedly, before the item collects the data with the container running state. Well, I managed to be faster, so now you will see what happens when it “fails” to bring the container back to running state. Here in the Actions, we can see that Zabbix executed the script three times (I also stopped the container 3 times fast enough!) after which the action sent a notification to the admin about a failure to bring the container up with restarts.

Fig 15. Problem event about stopped container exit code 137

I have received the message that the container restart was unable to bring the container to a running state and requires human interaction to fix this.

Fig 16. Problem event notification in email

Summary

Now you know how to plan ahead and make use of the built-in capabilities of Zabbix to solve the issue without human intervention (where possible) and only get notifications when the automatic remediation attempt fails.

The post Zabbix and the Docker API, Part 3: Control appeared first on Zabbix Blog.

Zabbix and the Docker API, Part 2: Adapt

2026-04-29 Janis Eidaks

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/zabbix-and-the-docker-api-part-2-adapt/32912/

In this blog post, I will show you how to create a template for monitoring your Docker server with only API calls (without the Zabbix agent 2). Instead of creating a template, templated items, LLD rules, and trigger prototypes from scratch, we will adapt them from the existing template “Docker by Zabbix agent 2.”

How does the Zabbix agent 2 do it?

If you are wondering how the Zabbix agent 2 collects the data, you can look into the source code and see the magic behind the scenes: https://github.com/zabbix/zabbix/blob/master/src/go/plugins/docker/metrics.go.

In short, it uses a Unix non-TCP socket, makes the Docker API requests on the host, and returns JSON objects. Hey, we already know how to use it ourselves from the previous blog post, right?

For the Zabbix agent 2 to work with the Docker template, it needs access to the Unix socket, either by adding the user Zabbix to the group: Docker or running the Zabbix-agent2 as root.

Fig 1. The Zabbix agent 2 Docker plugin source code

How we will do it

I can adapt this template, improvise whenever I encounter a non-existing metric, and overcome any technical challenge with effort! For the most part, in the template we have a few Zabbix agent items that collect data in bulk and a lot of dependent items (and dependent item prototypes from the LLD rules).

The path forward is quite straightforward – we will clone the template and replace the Zabbix agent item type with the HTTP agent type item and add additional parameters shown below. I will also add additional user macros on the template, including the Docker server IP address, port, CA, SSL certificate, and key file names so that these can be adjusted on the host level.

Fig 2. The workflow of the template modifications

Cloning the template and changing the item type

First, clone the template “Docker by Zabbix agent 2” and give it a new name: “Docker stats by HTTP.” Next, modify the Templated Zabbix agent type item configuration with the following parameters:

Fig 3. The modification of the templated item configuration

Modify “Docker by HTTP” template items:

● Modify item: Get containers
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/containers/json?all=true     
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● Modify item: Get data_usage
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/system/df
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● Modify item: Get images
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/images/json
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

●Modify item: Get info
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/info
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● Modify item: Ping
  ▪ Type HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/_ping
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}
♯ Preprocessing (additional first step)
  ▪ Boolean to Decimals

We also need to modify the LLD rule configuration. Change item type from Zabbix agent type to HTTP agent type:

Fig 4. The modification of LLD discovery rule: containers discovery

Modify LLD rule: Containers discovery
● Discovery rule
  ▪ Type: HTTP agent
  ▪ Key: docker.containers.discovery[true]
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/containers/json?all=true     
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}
  ▪ Update interval: 1h
● LLD macros
  ▪ {#ID}   $.Id
  ▪ {#NAME} $.Names.first()

Modify LLD rule: Images discovery
● Discovery rule
  ▪ Type: Dependent item
  ▪ Master item: item> Get images
● LLD macros
  ▪ {#ID}   $.Id
  ▪ {#NAME} $.RepoTags

After that, we will also make changes in the LLD rule “Containers discovery” by modifying a few existing item prototypes (Zabbix agent type) and adding a new item. Below are the item prototypes that require modification:

● In LLD rule Containers discovery, modify parameters for item prototype: Container {#NAME}: Get info
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/containers{#NAME}/json       
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● In LLD rule Containers discovery, modify item prototype: Container {#NAME}: Get stats
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/containers{#NAME}/stats?stream=false  
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● In LLD rule Containers discovery, modify parameters for item prototype: Container {#NAME}: CPU percent usage
  ▪ Type: Calculated
  ▪ Formula: last(//docker.container_stats.cpu_usage.total.rate["{#NAME}"])/last(//docker.container_stats.system_cpu_usage.total.rate["{#NAME}"])*last(//docker.container_stats.online_cpus["{#NAME}"])*100
♯ Preprocessing (delete preprocessing step JSONPath)

● In LLD rule Containers discovery, add new item prototype: Container {#NAME}: System CPU total usage per second
  ▪ Name: Container {#NAME}: System CPU total usage per second
  ▪ Type: Dependent item
  ▪ Key: docker.container_stats.system_cpu_usage.total.rate["{#NAME}"]
  ▪ Type of information Numeric (float)
  ▪ Master item    prototype > Container {#NAME}: Get stats

♦ Tags (name:value)        
  ▪ component:cpu
  ▪ container:{#NAME}      

♯ Preprocessing

  ▪ JSONPath       $.cpu_stats.system_cpu_usage
  ▪ Change per second
  ▪ Custom multiplier: 1.0E-9

Cloning the template (again) and making minor modifications

Next, I will clone the template “Docker statistics by HTTP” and give the copy a new name – “Docker containers by HTTP.” In the template “Docker containers by HTTP,” delete the LLD rule Images discovery; delete templated items; from the LLD rule “Containers discovery” rule, delete all prototype entities (item prototypes), and add a filter in the LLD rule (shown below):

In LLD rule Containers discovery rule, in Filter tab: add additional filter option
Filters [type of calculation: A and B and C]
   ▪ {#ID} matches {HOST.HOST}

I have also created a host group “Docker” where the discovered container hosts will be added. In the template “Docker statistics by HTTP” delete all item prototypes in the LLD rule “Containers discovery.” We will create a Host prototype in the LLD discovery rule “Containers discovery” – the parameters are shown below:

Host prototype in LLD rule: Containers discovery
  ▪ Host name:     {#ID}
  ▪ Visible name:  {#NAME}
  ▪ Templates:     Docker containers by HTTP

Fig 5. Host prototype settings in LLD rule: Containers discovery

Fig 6. The cloned and modified templates

Creating a host and linking the template

Now all that is left is to create a host and link a template: “Docker statistics by HTTP.” Do not forget to add the correct Docker IP address or DNS name in the user macro.

I have created a new host “Docker server,” linked a template, and modified the user macro for the Docker IP address. This host will collect Docker overall statistics. After the LLD discovery execution, the container names will be automatically discovered and container hosts will be created with a linked template.

Fig 7. The Discovered container hosts with linked templates

If for some reason you are monitoring multiple Docker instances, you could have the same container names discovered, which will lead to LLD errors, as there can’t be hosts with the same name (container ID) or visible name (container name). Quick solution – for each Docker instance, add a prefix to each container name. Another solution – don’t split the template into two parts, then the items will be discovered under the same host, and you will not have this issue.

The Docker server host shows general information about the Docker server’s overall state and status:

Fig 8. Docker server hosts the latest data

A host will be created automatically for each discovered container and will collect the container-specific performance metrics:

Fig 9. The container: zabbix-agent2 latest data

Summary

Now you and I know a little bit more about how Zabbix agent2 is collecting Docker metrics. This blog post has shown you how to improvise and adapt existing templates with different data collection methods. Zabbix is a very versatile tool that you can use in multiple ways to get the data if you have some technical constraints. The included template can also be used as is, or you can modify it to suit your needs.

The post Zabbix and the Docker API, Part 2: Adapt appeared first on Zabbix Blog.

Zabbix and the Docker API, Part 1: Inspect

2026-04-22 Janis Eidaks

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/zabbix-and-the-docker-api-part-1-inspect/32860/

In this blog post, I will show you how to configure Zabbix to securely gather Docker API metrics using the Zabbix HTTP agent item with certificate authentication. This guide will cover configuring the Docker API and the Zabbix server side to gather data more securely.

Getting the data to Zabbix from the Docker API

By default, Docker API uses a non-network socket for security reasons, and there are several valid reasons for this. It is not advised to expose your Docker environment over TCP to localhost, and even less to the internet. Exposing the Docker API without any security to the internet is just inviting hackers for free lunch, as anyone (bots included) who can access your Docker API will also be able to do malicious operations with it (make changes, launch malicious containers, try to take over your environment, and do a lot of harm in general) !

So, make sure to harden your environment’s security and use this guide at your own judgment. Also, set up your firewall so only the Zabbix server can access the Docker API port! By default, you can check if the Docker service is active and if you can get a response to the Docker API by running the curl command:

# systemctl is-active docker
# curl --silent --show-error --unix-socket /var/run/docker.sock http://localhost/info |jq

Fig 1. Output of the Docker API call in CLI

Generating the certificates for the Docker and Zabbix server

You can use the right tool for the job, such as an HTTP agent for the Docker API requests with proper certificate authentication. You will require the CA private key and CA certificate; private key and certificate for the Docker server; private key and certificate for the Zabbix server (for simplicity, you can generate all of them on the Docker server and copy the appropriate files to the Docker server and Zabbix server directories).

A guide you can follow to generate the certificate files is located here: https://docs.docker.com/engine/security/protect-access/#use-tls-https-to-protect-the-docker-daemon-socket.

Deploying the certificate files and configuring the services

On the Docker server, copy the CA and server certificate files to /etc/docker directory:

# cp -v {ca,server-cert,server-key}.pem /etc/docker

The Docker daemon also requires JSON configuration with additional settings (allow TCP/ Unix socket, TLS options):

# nano /etc/docker/daemon.json

{
  "hosts": ["tcp://0.0.0.0:2376","unix:///var/run/docker.sock"],
  "tls": true,
  "tlsverify": true,
  "tlscacert": "/etc/docker/ca.pem",
  "tlscert": "/etc/docker/server-cert.pem",
  "tlskey": "/etc/docker/server-key.pem"
}

We will have to add the Docker service override to remove the Unix socket from the Docker systemd service, then reload the daemon, and restart the Docker service.

# mkdir -p /etc/systemd/system/docker.service.d

# nano /etc/systemd/system/docker.service.d/override.conf

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd

# systemctl daemon-reload
# systemctl restart docker

On the Zabbix server side, create directories for certificate files. Then, copy the relevant certificate files from the Docker server. In my case, I generated all of the certificate files on the Docker host (replace docker in the scp command with IP/DNS name of the Docker server): ca.pem, client-cert.pem, client-key.pem, to their respective directories and change their permissions.

# mkdir -pv /etc/zabbix/ssl/{ca,certs,keys}
# scp root@docker:/root/dockercerts/ca.pem /etc/zabbix/ssl/ca/
# scp root@docker:/root/dockercerts/client-cert.pem /etc/zabbix/ssl/certs/
# scp root@docker:/root/dockercerts/client-key.pem /etc/zabbix/ssl/keys/
# chmod -v 0400 /etc/zabbix/ssl/keys/client-key.pem
# chmod -v 0444 /etc/zabbix/ssl/ca/ca.pem /etc/zabbix/ssl/certs/client-cert.pem
# chown zabbix:zabbix -R /etc/zabbix/ssl

Check if you can get data in the Zabbix server from the Docker server with HTTPS request (replace $HOST with your Docker server address):

# curl -sS https://$HOST:2376/info \
  --cert /etc/zabbix/ssl/certs/client-cert.pem \
  --key /etc/zabbix/ssl/keys/client-key.pem \
  --cacert /etc/zabbix/ssl/ca/ca.pem |jq

Fig 2. Executing the HTTPS request to the Docker server from the Zabbix server machine

If everything works so far, then it is time to modify the Zabbix server configuration file and specify the location of the certificate file directories. After that, restart the Zabbix server service.

# nano /etc/zabbix/zabbix_server.conf

SSLCertLocation=/etc/zabbix/ssl/certs
SSLKeyLocation=/etc/zabbix/ssl/keys
SSLCALocation=/etc/zabbix/ssl/ca

# systemctl restart zabbix-server

You will also need to copy the Docker CA file to the trusted CA directory and update the CA list.

# cd /
# cp /etc/zabbix/ssl/ca/ca.pem /etc/pki/ca-trust/source/anchors/
# update-ca-trust extract

Fig 3. The location of certificate files in the directories for each server

Configuring the monitoring in the Zabbix frontend

If you have read this far and decided that this is too much work or this approach is not feasible in your environment (company policy or some other technical limitation), don’t be discouraged so fast! There is another way to get the metrics without changing the Docker configuration, creating certificates, and configuring the Zabbix server config file – simply use an SSH agent-type item to gather the data.

To prepare for both approaches, I will create a host with multiple user macros, which will store the IP address, port, SSH user, SSH password, and SSL certificate information.

The easy way: SSH agent items

However, what to do if the company policy prohibits installing additional applications to gather data, such as the Zabbix agent (or changing Docker configuration settings, as in this case)? In this instance, you can use other, seemingly simpler ways to gather metrics, such as using the SSH agent item.

If the only tool you have is a hammer (SSH access), you tend to see every problem as a nail. The old adage “do not fix what is not broken” is still prevalent in this era! In that case, create an SSH agent-type item. Specify the IP address and SSH port in the item key, the username and password for the Docker host, and specify a command to gather the data. For those fields, I will use the previously defined user macros.

Here is an example of the SSH item configuration:

Host: Docker server items 
Item #1
  ▪ Name:          Get info ssh
  ▪ Type           SSH agent
  ▪ Key:           ssh.run[docker.infos,{$DOCKER.IP}]  
  ▪ Type of inf:   text
  ▪ Username:      {$SSH.USER}
  ▪ Password:      {$SSH.PASSWORD}
  ▪ Ex. script:    curl --unix-socket /var/run/docker.sock http://localhost/info

Fig 6. Example of SSH agent item configuration for executing a script on the Docker server

You can also test the item and obtain the same data in JSON format, shown in Fig. 1.

The right way: HTTP agent

For the other approach, we will be using an HTTP agent item to collect the data in bulk, using Docker API calls. For this, I don’t need to install the Zabbix agent on the Docker server. The authentication of this item will be performed using the certificates that have been copied over. Here are the important parameters in the item:

Host: Docker server items 
Item #1
  ▪ Name:             Get info
  ▪ Type              SSH agent
  ▪ Key:              docker.info    
  ▪ Type of inf:      text
  ▪ URL:              https://{$DOCKER.IP}:{$DOCKER.PORT}/info
  ▪ SSL verify peer: check
  ▪ SSL verify host: check
  ▪ SSL cert. file:  {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file:    {$SSL.KEY.FILE}

Do not forget to test the item (collected data should be the same as in Fig. 2) and add the item. If you have also encrypted the client private key (client-key.pem), you will also need to provide an SSL key password in the item configuration.

Fig 8. Example of the configured HTTP agent item

Fig 9. HTTP agent item collecting the data

Extracting the data

Now we can extract the important metrics by creating dependent items using the master item: Get info. Add a few dependent items to extract metrics, such as the total count, running, stopped, and paused containers. Item configuration parameters are given below the dependent item examples. The item “Containers running” parameter screenshots are shown below, together with the configuration parameters listed.

Fig 10. Dependent item tab to get the number of running containers

Tagging an item will also make your life easier for filtering when you have a legion of items.

Fig 11. Dependent item tag tab to get the number of running containers

In the preprocessing tab, we can use the JSONPath preprocessing step to extract the number of running containers from the master item.

Fig 12. Dependent item preprocessing tab to get the number of running containers

Docker Host items
● Item #1
  ▪ Name: 	Containers running	
  ▪ Type 		Dependent item
  ▪ Key: 		docker.containers.running	
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item	Docker: Get info
  ▪ Units: 	!containers
♦ Tags (name:value) 	
  ▪ component:containers	
♯ Preprocessing
  ▪ JSONPath  	$.ContainersRunning

● Item #2
  ▪ Name: 	Containers paused	
  ▪ Type 		Dependent item
  ▪ Key: 		docker.containers.paused	
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item	Docker: Get info
  ▪ Units: 	!containers
♦ Tags (name:value) 	
  ▪ component:containers	
♯ Preprocessing
  ▪ JSONPath  	$.ContainersPaused

● Item #3
  ▪ Name: 	Containers stopped	
  ▪ Type 		Dependent item
  ▪ Key: 		docker.containers.stopped	
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item	Docker: Get info
  ▪ Units: 	!containers
♦ Tags (name:value) 	
  ▪ component:containers	
♯ Preprocessing
  ▪ JSONPath  	$.ContainersStopped

● Item #4
  ▪ Name: 	Containers total	
  ▪ Type 		Dependent item
  ▪ Key: 		docker.containers.total	
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item	Docker: Get info
  ▪ Units: 	!containers
♦ Tags (name:value) 	
  ▪ component:containers	
♯ Preprocessing
  ▪ JSONPath  	$.Containers

Creating the trigger

I can also configure a trigger to receive a problem event in case some containers are not running. The screenshot of the trigger and parameter configuration is shown below.

Trigger
◘ Trigger 
  ▪ Name: 		Some containers are not running
  ▪ Operational data: 	Total: {ITEM.LASTVALUE1}, Running: {ITEM.LASTVALUE2}
  ▪ Severity: 		Warning
  ▪ Expression: 		last(/Docker server/docker.containers.total)last(/Docker server/docker.containers.running)
  ▪ PROBLEM event generation mode: Single
  ▪ OK event closes: All problems

Getting more data

Docker Engine also includes previous API versions. If no version of the API is specified in the URL, then the latest installed version will be used (using the API without a version is deprecated and will be removed in a future release). So even if you have the latest Docker installed (and you should always update to the latest version!), you can still use the older API calls by specifying the version (but once again, check what works).

Docker API offers several API calls that can be used to collect information about containers, images, container performance statistics, networks, volumes, or make changes to them.

Also, for more API calls, please explore this page: https://docs.docker.com/reference/api/engine/latest/.
As an example, I will create another item to gather specific container information. The item configuration will differ from the one in the example in Fig.8 with the following parameters: different URL, item name, and key.

Here is an example of the ULR field (replace {$CONTAINER} with the existing container name):

https://{$DOCKER.IP}:{$DOCKER.PORT}/containers/{$CONTAINER}/json

Fig 14. HTTP agent item to get low-level information about a specific container: tomcat

You can also get the container performance data with a different URL. The item configuration will differ from the one in an example in Fig.8 with the following parameters: URL, item name and key. Here is an example of ULR field (replace {$CONTAINER} with the existing container name):

https://{$DOCKER.IP}:{$DOCKER.PORT}/containers/{$CONTAINER}/stats?stream=false

Fig 15. HTTP agent item to get performance information about a specific container: zabbix-server-mysql

Testing the trigger

We can test if the data returned by the Docker API is as it seems, right? I have five containers created using the ‘docker run’ command, and one using the ‘docker compose’ command. Let’s stop the container made from the ‘docker run’ command and check if it will be reflected in the collected metrics.

Fig 16. The latest item data when stopping a Docker Compose container

As you can see in Figure 13, the stopped container shows up in the metrics collected by Zabbix through Docker API and in the Docker CLI. The Docker host item shows 1 stopped container and 5 running containers; the total number of containers is 6.

If you use the command “docker compose down” instead, the container will be stopped and removed altogether. That means, the total number of containers will also decrease by one, along with its status, as shown in Fig. 17. Therefore, make sure you understand what each command does and how it will impact your monitoring data.

Fig 17. The latest item data when using Docker Compose down for a container

In summary

Now you know more about how to collect the data from Docker using HTTP requests. Similar approaches can also be used to collect data from other applications through an API. You can select what metrics you want to extract, create triggers, graphs, or make a template if you wish.

The post Zabbix and the Docker API, Part 1: Inspect appeared first on Zabbix Blog.

Detect Issues in Your Zabbix Instance Before It’s Too Late

2026-03-25 Janis Eidaks

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/detect-issues-in-your-zabbix-instance-before-its-too-late/32741/

In this blog post, I will show you how to detect performance issues in your Zabbix instance – in advance!

You might be using Zabbix to monitor your infrastructure, devices, and applications, but are you also monitoring your own instance? It might seem unnecessary – after all, what’s there to monitor, right? Your instance just works, so everything is good. What else is needed?

Remember though, if your Zabbix database system runs out of disk space, data collection will come to a halt. If the data collectors are insufficient, the collected data will be inconsistent, and this will also affect the problem detection.

If you run out of cache space on your Zabbix server, depending on which cache is affected, your Zabbix server might crash immediately or experience degraded performance. A lot of things can go wrong, and you need to stay ahead of them! Here’s how.

Tune your database

If you are using the default settings for your database, you are missing out on significant performance improvements that are just unused! Your actual instance performance is tied to the database performance. If the database performance is low, you will have a degraded Zabbix monitoring experience as well.

Do at least minimal fine-tuning, only change the settings you understand: read the documentation, check the official Zabbix blogposts, Zabbix community forum, and perform testing. Of course, you can use every tool at your disposal to make it work, such as AI, but always test the settings in the test environment.

The database tuning is a complex task. Initial parameters that you could tune for the MySQL DB are these:

innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
optimizer_switch=index_condition_pushdown=off
innodb_buffer_pool_size= ~75% of RAM if only DB engine running or less if shared with other applications

For a PostgreSQL database, you can use online tuner PGTUNE for initial configuration:

https://pgtune.leopard.in.ua/

Monitor the Zabbix database

It is important to monitor your database. Zabbix offers several out-of-the-box options to monitor the most popular databases through different methods: either by Zabbix agent or Zabbix agent2, by ODBC checks, using Zabbix Java gateway or by HTTP checks. If an issue is detected, you will get a corresponding problem event. Don’t forget to manually update the old Zabbix templates to the current version after the Zabbix server upgrades.

Fig 1. Some of the available out-of-the-box templates for database monitoring

Of course, depending on the approach you have selected to monitor the database, you will need to do some additional steps for that to work. More information on how to configure it is available on the Zabbix integration page.

Fig 2. Example of the configuration required to monitor the MySQL database with Zabbix agent2

Monitor the Zabbix server

The next thing you should check is the Zabbix server host dashboards. In new instances, the Zabbix server host has already been included out of the box with two templates: Zabbix server health and Linux by Zabbix agent. If such a host has not been retained for some reason, now it’s time to create it and start monitoring your Zabbix server.

The Zabbix server health template uses Zabbix internal items that do not require any interface. The Linux by Zabbix agent template does require a running Zabbix agent on the Zabbix server system in order to gather the OS related metrics.

Fig 3. Zabbix server host with linked templates

Check the current state of your Zabbix server

Once you have such a host, go to the menu Monitoring > Hosts and use the main filter to find the Zabbix server host and select its Host Dashboards.

Select the Zabbix server health dashboard. Below, you will see the following pages under it – Performance, Processes, and Statuses.

Fig 5. Zabbix server health dashboard page: Performance

Check the cache utilization

In the Performance page, you can see the usage of Zabbix server caches. You should make sure that all caches except the history cache are at least ~ 50 % free. Technically, you can make the caches as large as possible; at worst, they will just be under-utilised. So, adjust the cache sizes accordingly.

Consequences of running out of configuration cache

If you add a lot of hosts in an automated way and have a relatively small or default configuration cache size [configcache], you could fill this cache quickly. The consequences of it are:

The Zabbix server will crash
The Zabbix server will be unable to start
The Zabbix server will not collect any data

You will also see a warning message in the Zabbix frontend:

Fig 6. Zabbix server health dashboard page when running out of config cache

If the config cache does not fill up instantly, the problem event will be generated shortly after, and matching action operations will be executed while the Zabbix server is still running, for example, notifying admins about the issue. In the screenshot below, you can see that one action operation step was executed before the server crashed.

When a Zabbix component is not working as expected, your best source of information is the log file, as it informs you about the issues. Here is the error message in the Zabbix Server log file below.

Fig 8. Zabbix server log file error: out of memory for config cache

The solution is very simple: just increase the configuration cache size (two times or more) and restart the Zabbix server. If you expect a significant increase in hosts in the near future, you can be more generous and allocate more memory. My current Zabbix server is monitoring approximately 400 hosts.

Fig 9. The system information of my Zabbix server

Consequences of running out of value and history cache

So, what happens if you run out of value cache? Zabbix server performance will degrade, and the frontend will become noticeably less responsive. Why is that? Value cache stores item values used for calculated items and evaluating triggers. Now, for each trigger calculation that does not contain an item metric in the history cache will be retrieved directly from the database.

Fig 10. Zabbix server health dashboard performance page for cache usage

The history cache stores historical data that will be written to the database. If it’s mostly full, it means you might have issues with your database performance – the Zabbix server is unable to write data fast enough to the database. This can trigger a cascading performance degradation with a negative feedback loop. In my case:

A full value cache leads to additional DB read queries
DB performance drops, which leads to slow historical data writes to the database
The history cache also starts to fill up
The data collection is delayed due to the full history cache
As more data is collected, database read queries retrieve more data, progressively worsening the cycle

Technically, it does not require your value cache to be 100% full to have this issue – if a lot of triggers use a long-time interval for evaluation, you could have a situation where your value cache is only 85% or 90% full, but the Zabbix server is unable to fit the required item history records in available memory.

The issue with running out of value cache will also be logged in the Zabbix server’s log file.

Fig 11. The Zabbix server log file with value cache error

The solution to this issue is simple: increase the value cache size and restart the Zabbix server.
If you monitor your Zabbix server with the health template, problem events will be automatically generated when:

Value cache works in a low memory mode
History cache utilization exceeds 75 %

Fig 12. The generated problem events about the value cache issue

The issue with the Value cache working in low memory mode can also be seen in the graph below. Here you can see how many historical item values were present in cache, and how many had to be retrieved from the database directly.

Due to the terrible performance of the untuned database, when my history write cache fills up the data collectors are throttled, causing a pileup of delayed item collection.

Slow database queries will appear in the Zabbix server log file.

Fig 15. The Zabbix server log file with slow query errors

The result of cache tuning and database tuning

Increasing the value cache only partly solved one issue. After database tuning, database performance has improved significantly:

The history cache is now empty
No more value cache misses
No more delayed items

Fig 16. The Cache usage, server performance, and value cache effectiveness graphs

After the Database tuning, the agent poller process and history syncer utilization also decreased to a low level.

Fig 17. Data collector and internal process utilization graphs

Tune the Zabbix server configuration parameters

Check the Processes page in the Zabbix Health dashboard and adjust the parameters accordingly. Only adjust the parameters that you understand. Changing the parameters arbitrarily can lead to the following:

Wasted resources without effective performance improvement
Reduced Zabbix server performance
Zabbix server crashes

For the data collectors, generally you require only a relatively small number of asynchronous data collectors, as they are very efficient, relatively larger number of synchronous data collectors. The graphs showing the utilisation of the gathering processes are extremely useful for determining which ones need to be increased – if they are close to 100% utilized, it’s now time for you to take action and add more.

Pitfalls of misconfiguration

Now, regarding the pitfalls of misconfiguration or lack of tuning. Here is a scenario: installed the Zabbix components, MySQL database without any configuration tuning, except the configuration cache to avoid the Zabbix server crashing immediately. The Zabbix server is monitoring around ~400 hosts. The Zabbix agent poller process and history syncers are utilized close to 100%, like in the Fig.17 before the tuning.

You might think that increasing both of these processes would improve the situation, for example, by doubling the count of them: more parallel agent processes should collect more data, and the more history syncers should write more data to the database.

After restarting the Zabbix server and checking the graph, both processes are close to 100% busy and the metric collection is significantly delayed. This is much worse.

Fig 18. Async data collector and internal process utilization graphs

By quadrupling both processes, the result is even worse, with significantly delayed item value collection.

Fig 19. Async data collector and internal process utilization graphs

So, what is happening behind the frontend? Just increasing the number of agent poller collectors and history syncers results in even worse performance. Seems counterintuitive, right: more data collectors should mean more data will be collected, and more history syncers – should allow more data to be written in parallel to the database.

However, increasing the data collector count in this specific situation will just make things much worse: you can collect more data at the same time, but will still face the same bottleneck: the database. Increasing the history syncers in this case makes the situation much worse, as more simultaneous queries to the database force it to slow down even further. So once again, tune your database engine and get more performance out of it.

Summary

You should monitor all of your Zabbix components and react when issues occur. Also make sure that you receive the notifications in your preferred media type, so you can act immediately. The complete list of what to monitor is more extensive, but this blog post should provide you with some examples and inspiration. It is always a good idea to react proactively rather than deal with the issues after they occur.

The post Detect Issues in Your Zabbix Instance Before It’s Too Late appeared first on Zabbix Blog.

Podman Container Monitoring with Prometheus Exporter, part 2

2025-06-12 Janis Eidaks

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/podman-container-monitoring-with-prometheus-exporter-part-2/30538/

In the first part of this post, we explored how to get data with HTTP agent from the Prometheus Podman exporter and use the same item data for the Podman pods Discovery rule as well as item and trigger prototypes. In part 2 of the same series, we’ll learn how to discover and monitor Podman containers.

Table of Contents

Creating a template discovery rule

I will create another discovery rule for container discovery. This discovery rule is also based on the same item [Podman info] in the template – Podman containers by HTTP and Prometheus (you can check part one of this series to find out how to configure it). The parameters of the discovery rule are shown below. This discovery rule will allow us to discover the pod name and ID.

Template: Podman containers by HTTP and Prometheus

▲ Discovery rule
  ▪ Name:                   Container discovery
  ▪ Type                    Dependent item
  ▪ Key:                    training.containers.discovery
  ▪ Master item             Podman containers by HTTP and Prometheus: Podman info
  ▪ Delete lost resources  After 10d
  ▪ Disable lost resources Immediately
♯ Preprocessing
  ▪ Prometheus to JSON     podman_container_info
♦ LLD Macros
  ▪ {#CONTAINER.ID}        $.labels.id
  ▪ {#CONTAINER.NAME}      $.labels.name

Fig 1. Discovery rule: Container discovery

Fig 2. Discovery rule: Container discovery preprocessing tab

Fig 3. Discovery rule: Container discovery LLD macros tab

Next, different dependent item prototypes are created in this container discovery rule. As the Prometheus Podman exporter provides a lot of different metrics about the containers, I will create multiple such items: state, health, creation date, input/output network traffic information, and so on. So, check out what metrics can be acquired and use what is relevant for you.

You can also add a description of each item prototype. I am interested only in metrics with the discovered container ID macros, and I am not interested in what values are for the other fields, such as pod_id, pod_name, so I use ~”.*”, which matches any value. I will show the item prototype configuration screenshots of one of the item prototypes.

These item prototypes are similar to each other, with some minor differences, such as Prometheus patterns, or in some cases, with a different master item (item prototype as master item).

Fig 4. Discovery rule preprocessing step: Prometheus to JSON with pattern podman_container_info

Fig 5. Discovery rule LLD macros: assigning relevant JSONPATH to LLD macros

Creating a template discovery rule: Item prototypes

After the containers have been discovered, we have to create item prototypes. These prototypes will also be dependent item prototypes and will use the same item as the discovery rule: Podman info. Prometheus Podman exported returns a lot more metrics for the containers than it did for the pods.

You can get container metrics such as container health, state, creation date, disk read/write, memory usage, network usage, and more. In this blog post, I have added most of them, so check what metrics are relevant to your monitoring needs and start monitoring.

Fig 6. Low-level discovery rule and item prototypes based on the same item.

The screenshots of the item prototype is shown below.

Fig 7. Container state item prototype tab

Fig 8. Container state item prototype tag tab

Fig 9. Container state item prototype preprocessing tab

Remember, you can also test these item prototypes in the preprocessing step – just copy the Prometheus exporter data and set the relevant macro to value you want to check.

The configuration parameters of the item prototypes are shown below. There are a lot of metrics you can monitor, but remember to monitor what is relevant and necessary for you.

Template: Podman containers by HTTP and Prometheus; Discovery rule: Container discovery

● Item prototype #1
  ▪ Name: 		Container health: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.health[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (float)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 
♦ Tags (name:value) 	
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:health		
♯ Preprocessing
  ▪ Prometheus pattern 	podman_container_health{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #2
  ▪ Name: 		Container state: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.state[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (float)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		
♦ Tags (name:value)  			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:state		
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_state{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #3
  ▪ Name: 		Created at: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.created[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		unixtime
♦ Tags (name:value) 		
  ▪ Container:{#CONTAINER.NAME}	
  ▪Metric:created		
♯ Preprocessing
  ▪ Prometheus pattern 	podman_container_created_seconds{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #4
  ▪ Name: 		Disk read per second: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.disk.read[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		B
♦ Tags (name:value) 	
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:disk_read		
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_block_output_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value
  ▪ Change per second

● Item prototype #5
  ▪ Name: 		Disk write per second: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.disk.write[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		B
♦ Tags (name:value) 	
  ▪ Container:{#CONTAINER.NAME}	 
  ▪ Metric:disk_write		
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_block_input_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value
  ▪ Change per second

● Item prototype #6
  ▪ Name: 		Exit code: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.exit_code[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (float)
    ▪ Master item	Podman containers by HTTP and Prometheus: Podman info
▪ Units: 			
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:exit_code	
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_exit_code{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #7
  ▪ Name: 		Image tags: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.image.tags[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Character
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 			
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:tag
♯ Preprocessing
▪ Prometheus pattern podman_container_info{id="{#CONTAINER.ID}",image=~".*",name=~".*",pod_id=~".*",pod_name=~".*",ports=~".*"} label image
  ▪ Regular expression	\.*(\/.\w.*)	\1

● Item prototype #8
  ▪ Name: 		Memory usage: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.mem[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		B
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:mem		
♯ Preprocessing
  ▪ Prometheus pattern podman_container_mem_usage_bytes{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #9
  ▪ Name: 		Network input dropped: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.in.drop[{#CONTAINER.NAME}]
  ▪ Type of inf: Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		packets
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_in_drop		
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_net_input_dropped_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #10
  ▪ Name: 		Network input errors: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.in.errors[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_in_err		
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_net_input_errors_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #11
  ▪ Name: 		Network input total: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.in.total[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		B
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_in_tot
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_net_input_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #12
  ▪ Name: 		Network input per second: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.in.change[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (float)
  ▪ Master item		prototype - Network input total: [{#CONTAINER.NAME}] 
  ▪ Units: 		Bps
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_in_change
♯ Preprocessing
  ▪ Change per second

● Item prototype #13
  ▪ Name: 		Network output dropped: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.out.drop[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_out_drop	
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_net_output_dropped_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #14
  ▪ Name: 		Network output errors: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.out.errors[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_out_err	
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_net_output_errors_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #15
  ▪ Name: 		Network output total: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.out.total[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		B
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_out_tot	
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_net_output_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #16
  ▪ Name: 		Network output per second: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.net.out.change[{#CONTAINER.NAME}]
  ▪ Type of inf: 	Numeric (float)
  ▪ Master item		prototype - Network output total: [{#CONTAINER.NAME}]
  ▪ Units: 		Bps
♦ Tags 			 
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:net_out_change
♯ Preprocessing
  ▪ Name			Change per second

● Item prototype #17
  ▪ Name: 		Rootfs size: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.rootfs.size[{#CONTAINER.NAME}]
  ▪ Type of inf: Numeric (unsigned)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		B
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:rootfs
♯ Preprocessing
  ▪ Prometheus pattern	podman_container_rootfs_size_bytes{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

● Item prototype #18
  ▪ Name: 		Total system CPU time: [{#CONTAINER.NAME}]
  ▪ Type 		Dependent item
  ▪ Key: 		container.cpu.time
  ▪ Type of inf: 	Numeric (float)
  ▪ Master item		Podman containers by HTTP and Prometheus: Podman info
  ▪ Units: 		s
♦ Tags 			
  ▪ Container:{#CONTAINER.NAME}	
  ▪ Metric:sys_time
♯ Preprocessing
  ▪ Prometheus pattern: podman_container_cpu_system_seconds_total{id="{#CONTAINER.ID}",pod_id=~".*",pod_name=~".*"} value

Creating a template discovery rule: Trigger prototype

I have created a user macro {$CONTAINER.RUNNING.STATE} on the template with a value of 2, which corresponds to the containers running state. After that, create a trigger prototype to check if the container is in different state other than running.

Template: Podman containers by HTTP and Prometheus; Discovery rule: Container discovery

◘ Trigger prototypes
  ▪ Name:               Container [{#CONTAINER.NAME}] state has changed from running
  ▪ Severity:           Warning
  ▪ Expression:         last(/Podman containers by HTTP and Prometheus/container.state[{#CONTAINER.NAME}])<>{$CONTAINER.RUNNING.STATE}
  ▪ PROBLEM event generation mode: Single
  ▪ OK event closes: All problems

So, once all of this is done, and some container status changes from running and or pod status also changes from running, you will get a problem event.

Fig 10. Generated problem events when the podman pod and container change states.

Technically, I could also create a trigger for container health; however, as all of the received container values for me are -1 (meaning unknown) it makes little sense to make a trigger that will fire right away. You can also add additional item/trigger prototypes in the template. If everything is set up as expected, you should see something like the screenshot below after the LLD rule execution.

Fig 11. Example of the mysql-server container and zabbix pod item values.

Summary

Now, you can monitor both Podman pods and containers using both blog posts of this series. We used the same template item for both the container LLD and item prototypes from the first part of this post.

The post Podman Container Monitoring with Prometheus Exporter, part 2 appeared first on Zabbix Blog.

Podman Container Monitoring with Prometheus Exporter, part 1

2025-06-10 Janis Eidaks

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/podman-container-monitoring-with-prometheus-exporter-part-1/30513/

In part one of this blog post, I will show you how to monitor Podman pods using HTTP agent item to retrieve data from the Prometheus Podman exporter. Let’s get started!

Table of Contents

Installing and checking Prometheus Podman exporter

First, you will need to install and enable the Prometheus Podman exporter (my OS is CentOS Stream release 9). Then, check that the service is active and running.

# dnf install -y prometheus-podman-exporter

# systemctl enable prometheus-podman-exporter –now

# systemctl status prometheus-podman-exporter

You can check that you are getting the data from the exporter with either the curl command from the machine/VM where the Prometheus podman exporter is installed and started:

# curl http://localhost:9882/metrics

Fig 1. Output of Prometheus podman exporter in CLI

Or through the browser (replace abc with the machine’s IP/DNS ): abc:9882/metrics.

Fig 2. Output of Prometheus Podman exporter in browser

A line starting with # is a comment and contains an explanation regarding the metric; in this case, podman_container_block_input_total will return data in bytes. In Figure 2, after the comments, you can see several podman_container_block_input_total metrics, one for each container, with different container IDs, pod IDs, and pod names listed in each metric. The metric’s value is displayed on the right side after curly brackets.

Creating a template and template items

Next, I will create a template Podman containers by HTTP and Prometheus where I will put all of the entities (everything will be created on the template). In the template, I will create an item Podman info, which will gather all of the necessary data at defined intervals. This approach will be convenient from a data collection standpoint as the same item data will be used for LLD and item prototypes. During testing, you can set “History” to store data for some time, and when everything is working as expected, then set “History” not to keep any data. This item will be used for the Low-Level Discovery rule and the item prototype.

The item Podman info parameters are as follows:

Template: Podman containers by HTTP and Prometheus

○ Item
  ▪ Name:         Podman info
  ▪ Type          HTTP agent
  ▪ Key:          podman.info
  ▪ Type of inf   text
  ▪ URL           http://{HOST.CONN}:9882/metrics
  ▪ Request type  GET
  ▪ Update int.   5m
  ▪ Req status c. 200
  ▪ History       Do not store
◊ Tags
  ▪ Podman:raw

At this moment, this item will contain just raw data, without any preprocessing steps applied. The IP address will be taken from any host interface added to the host. You will get an error message if the host has no interface.

Fig 4. Error on the host with the linked template without any interface

If you do not want to add an interface to the host, you can define a user macro on the template level and use that user macro in the items URL. After adding the template to the host, just modify the user macro value on the host to correct IP/DNS name.

Fig 6. Template item for data gathering with user macro instead of built in macro from host interface

I can also create an item to determine the number of containers created. I can count specific Prometheus pattern occurrences in the master item to determine this. For this, I will use the podman_container_state parameter. Likewise, I could use different parameters, such as podman_container_info, and count the occurrences of such a pattern. The parameters of the item container count:

Template: Podman containers by HTTP and Prometheus

○ Item
  ▪ Name:         Container count
  ▪ Type          Dependent item
  ▪ Key:          container.count
  ▪ Type of inf   Numeric (unsigned)
  ▪ Master item   Podman containers by HTTP and Prometheus: Podman info
◊ Tags
  ▪ Containers:total
♯ Preprocessing
  ▪ Prometheus pattern     podman_container_state     count

Fig 7. Template item preprocessing step for counting the total number of containers

Creating a Discovery rule in template

Next, the LLD rule will be created to discover Podman pods. It will be a dependent LLD rule based on a Podman info item with a preprocessing step to convert the Prometheus pattern data to JSON format. The caveat is that the LLD discovery will be executed as frequently as the data is received for the item. If there are a lot of hosts with such a template, there will be a lot of LLD processes executed, which can put a strain on your Zabbix instance.

To rectify this issue, I will add a preprocessing step: discard unchanged with heartbeat (as there are no dynamic parameters in the extracted pattern, otherwise we would need to filter out dynamically changing information). For LLD discovery, the recommended interval is around 1h. Additionally, LLD macros will be created from selected JSNOPath variables. The parameters of the LLD rule are shown below.

Template: Podman containers by HTTP and Prometheus

▲ Discovery rule
  ▪ Name:                   POD discovery
  ▪ Type                    Dependent item
  ▪ Key:                    training.pod.discovery
  ▪ Master item             Podman containers by HTTP and Prometheus: Podman info
  ▪ Delete lost resources  After 10d
  ▪ Disable lost resources Immediately
♯ Preprocessing
  ▪ Prometheus to JSON     podman_pod_info
  ▪ Discard unchanged with heartbeat 1h
♦ LLD Macros
  ▪ {#POD.ID}              $.labels.id
  ▪ {#POD.NAME}            $.labels.name

Fig 9. Discovery rule: Pod discovery preprocessing tab

Fig 10. Discovery rule: Pod discovery LLD macros tab

The block diagram below will show how the data is transformed. First, a preprocessing step is applied to the data to convert the Prometheus pattern to JSON format, as all data for LLD must be supplied in JSON format.

In the example below, the matching queried pattern is returned in JSON format after this preprocessing step.

Fig 11. Discovery rule preprocessing step: Prometheus to JSON with pattern podman_pod_info

After the preprocessing step, we can assign specific JSONPATH values to LLD macros.

Fig 12. Discovery rule LLD macros: assigning relevant JSONPATH to LLD macros

Creating a template Discovery rule: item prototypes

Now that we have discovered the macros we are interested in, the discovered macros can be used for further prototype (ITEM/HOST/TRIGGER) creation. In this example, I am using the same master item for LLD discovery and the dependent item prototypes, because it is convenient for me, and all the information is available in one item. But usually, there are scenarios where you have to use one item’s data for discovery and the data of another item for populating the prototype values.

In this case, I am interested in the pod ID, when the pod was created, the number of containers in the pod, and the state of the pod. Therefore, I will create the item prototypes and use the LLD macro in the name, key, and preprocessing step. Zabbix will cycle through the discovered LLD macro values and create the items based on the prototype by replacing the LLD macro with discovered values. Although you can set matching item prototype names (which will be confusing), you still have to use the LLD macro in the item key so that different item keys are generated – otherwise, you will get an error regarding duplicate keys. The item prototype parameters are given below.

Fig 13. Low-level discovery rule and item prototypes based on the same item.

Template: Podman containers by HTTP and Prometheus; Discovery rule: POD discovery

○ Item prototype #1
  ▪ Name:         POD ID: [{#POD.NAME}]
  ▪ Type          Dependent item
  ▪ Key:          pod.id[{#POD.NAME}]
  ▪ Type of inf   Character
  ▪ Master item   Podman containers by HTTP and Prometheus: Podman info
♦ Tags
  ▪ Metric:ID
  ▪ Pod:{#CONTAINER.NAME}
♯ Preprocessing
  ▪ Prometheus pattern     podman_pod_containers{id="{#POD.ID}"}          label    id

○ Item prototype #2
  ▪ Name:         POD state: {#POD.NAME}
  ▪ Type          Dependent item
  ▪ Key:          pod.state[{#POD.NAME}]
  ▪ Type of inf   Numeric (float)
  ▪ Master item   Podman containers by HTTP and Prometheus: Podman info
  ▪ Value mapping POD state
♦ Tags
  ▪ Metric:state
  ▪ Pod:{#CONTAINER.NAME}
♯ Preprocessing
  ▪ Prometheus pattern     podman_pod_state{id="{#POD.ID}"}      value

○ Item prototype #3
  ▪ Name:         POD created at: [{#POD.NAME}]
  ▪ Type          Dependent item
  ▪ Key:          pod.created[{#POD.NAME}]
  ▪ Type of inf   Numeric (unsigned)
  ▪ Units         unixtime
  ▪ Master item   Podman containers by HTTP and Prometheus: Podman info
♦ Tags
  ▪ Metric:created
  ▪ Pod:{#CONTAINER.NAME}
♯ Preprocessing
  ▪ Prometheus pattern     podman_pod_created_seconds{id="{#POD.ID}"}     value

○ Item prototype #4
  ▪ Name:         POD container count: [{#POD.NAME}]
  ▪ Type          Dependent item
  ▪ Key:          pod.count[{#POD.ID}]
  ▪ Type of inf   Numeric (unsigned)
  ▪ Master item   Podman containers by HTTP and Prometheus: Podman info
♦ Tags
  ▪ Metric:count
  ▪ Pod:{#CONTAINER.NAME}

On the template, I have also created a value map for deciphering the numerical pod state codes to text strings for better clarity.

Fig 14. Value mapping for the POD state item

Here are some screenshots of the POD state item prototype, shown below.

Fig 15. POD state item prototype: item prototype tab

Fig 16. POD state item prototype: tag tab

Fig 17. POD state item prototype: preprocessing tab

Creating a template Discovery rule: trigger prototype

We can also create a trigger prototype to generate an alert if there is something wrong with the pod. I have created a user macro {$POD.RUNNING.STATE} on the template with a value of 4, which corresponds to the running state.

Template: Podman containers by HTTP and Prometheus; Discovery rule: POD discovery

◘ Trigger prototypes:
  ▪ Name:               POD [{#POD.NAME}] state has changed from running
  ▪ Severity:           Warning
  ▪ Expression: last(/Podman containers by HTTP and Prometheus/pod.state[{#POD.NAME}])<>{$POD.RUNNING.STATE}
  ▪ PROBLEM event generation mode: Single
  ▪ OK event closes: All problems

Fig 18. Trigger prototype based on POD state item value

Once you link the template to the host and execute the LLD rule, you should start seeing the Podman pods ( if you have them), similar to the screenshot below.

Fig 19. Latest data for the host with the linked template

Summary

This blog post shows how to get data with HTTP agent from Prometheus Podman exporter and use the same item data for the Discovery rule as well as item and trigger prototypes. Check out part 2 of this series to find out how to discover and monitor Podman containers.

The post Podman Container Monitoring with Prometheus Exporter, part 1 appeared first on Zabbix Blog.

Deploying Zabbix Components with Docker and Docker Compose

2025-04-08 Janis Eidaks

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/deploying-zabbix-components-with-docker-and-docker-compose/30025/

Installing Zabbix from packages can feel overwhelming, due to the availability of different configuration options. The detailed and comprehensive documentation certainly helps to check the purpose of these multiple options, what values can be set in their fields, and if one is required for your planned deployment. There are quite a few official Zabbix blog posts about Zabbix in containers, and this post is aimed at showcasing how additional Zabbix components can be easily set up in a docker environment, along with docker run and docker compose examples.

Table of Contents

For those who would prefer to use Zabbix in a containerized environment such as Docker, or who want to try out Zabbix quickly, this guide is for you (you can also check out the other Zabbix Docker blog posts). You can also mix and match Zabbix components installed from packages or built from source with those running in containers.

Please follow the official guide on how to set up the docker here.

To better understand the Zabbix architecture for those who are trying out Zabbix for the first time, I will give you an overview that should make it much easier to follow and understand Zabbix.

Zabbix consists of 3 main components (the bare minimum to get started):

Zabbix Server – responsible for everything related to data collection, trigger evaluation, event generation, and alerting.
Zabbix Frontend – responsible for the configuration (modifying or changing the configuration of the monitoring targets) and visualization (dashboards, graphs, tables, and widgets).
Database – this is where the Zabbix configuration and monitoring history data are stored.

You can monitor your targets with the bare minimum setup; however, more comprehensive and complete monitoring can be achieved by using the C-based Zabbix-agent or GO-based Zabbix-agent2 in combination with templates, user parameters, and more. To set up the minimum necessary Zabbix components, you can use this example in the guide.

There are also official guides available on the Zabbix documentation page (for both: the docker run and docker compose) or the Docker/Github.

As of this writing , these official Zabbix docker components are available from the docker hub page:

Zabbix Server (with MySQL/PostgreSQL database)
Zabbix Proxy (with MySQL/SQLite3 database)
Zabbix Frontend (Apache/Nginx with MySQL/PostgreSQL DB)
Zabbix Agent (TLS encryption)
Zabbix Agent2 (TLS encryption)
Zabbix Java Gateway
Zabbix SNMP traps
Zabbix Web Service

Tags are used to select which OS container an image will be based on, as well as which Zabbix component version you wish to employ. If you only specify tag value – latest, you will get the latest Zabbix version based on the Alpine Linux. The images based on Linux Alpine are more lightweight than the other distros.

When something does not work as expected or fails, check the container error logs! This will be useful for debugging purposes and will help to narrow down the cause of an issue. Additionally, when debugging you can also specify additional options, such as specific lines of log, timestamp since or until, or following the log file content.

# docker logs --tail 50 container_name_or_id

    --details        Show extra details provided to logs
-f, --follow         Follow log output
    --since string   Show logs since timestamp (e.g. "2013-01-02T13:23:37Z") or relative (e.g. "42m" for 42 minutes)
-n, --tail string    Number of lines to show from the end of the logs (default "all")
-t, --timestamps     Show timestamps
    --until string   Show logs before a timestamp (e.g. "2013-01-02T13:23:37Z") or relative (e.g. "42m" for 42 minutes)

In some rare cases, when there is a container issue (everything else is correct, worked before, etc.), restarting the docker service can sometimes solve the issue.

So, what is different if you have only used Zabbix installed from packages? The examples below illustrate the differences in configuration options based on different Zabbix deployment methods: a) package-based/compiled installation, b) docker run command, and c) docker compose file example. First of all, you will have to specify environment variables in the docker run command or docker compose file. The list of available environment variables for each docker image is available in both docker hub and Github.

A). Package-based config

# vi /etc/zabbix/zabbix_server.conf
...
DBName=zabbix
DBUser=zabbix_usr
DBPassword=zabbix_pwd
...

B).Docker run config

docker run --name zbxsrv -t \
...
-e MYSQL_DATABASE=zabbix\
-e MYSQL_USER=zabbix_usr \
-e MYSQL_PASSWORD=zabbix_pwd\
...

C). Docker compose config

# vi /../...yaml
...
  environment:
   MYSQL_DATABASE=zabbix
   MYSQL_USER=zabbix_usr
   MYSQL_PASSWORD=zabbix_pwd

The environment variables are represented as key-value pairs, e.g., VAR=VAL. The values can optionally be unquoted or double-quoted. If some environment variable value contains special characters, you will need to escape them. To properly escape them, check out the docker documentation page.

You can create custom, user-defined networks to connect multiple containers to the same network. On such networks, containers can resolve each other by name or alias. If needed, you can assign a specific IP address to a container (if the address is already used, you will get an error).

# docker network create --subnet 172.20.0.0/16 --ip-range 172.20.240.0/20 zabbix-net

Docker run

In this section, we have an example of docker run commands for two Zabbix components: Zabbix proxy and Java gateway. When using custom, user-defined networks, you can use container names for communication between containers instead of using IP addresses. Here, instead of defining the IP address for Zabbix Java gateway, the container name is used. You can set a static IP address for your container or let docker do it for you, but confirm if the change of the IP address will not cause issues in case your container gets a different IP address. This can become an issue if you use an IP address in some configuration fields instead of a container name.

A lot of parameters are specified using environment variables with the option -e. Also, 3 different ports are exposed on your host machine. To keep the SQLite3 database file upon container deletion, the container directory containing database file is mounted to host directory (the proxy DB is usually used as a buffer storage before sending data to Zabbix server and usually is not used to store data beyond the moment when the data is sent).

docker run --name zabbix-proxy-active-01 \
-e ZBX_HOSTNAME="Zabbix-proxy-active-01" \
-e ZBX_SERVER_HOST=46.101.140.98 \
-e ZBX_PROXYMODE="0" \
-e ZBX_JAVAGATEWAY_ENABLE=true \
-e ZBX_JAVAGATEWAY=zabbix-java-gateway-proxy \
-e ZBX_JAVAGATEWAYPORT=10052 \
-e ZBX_STARTJAVAPOLLERS=5 \
--network=zabbix-net \
-e ZBX_LISTENPORT=10101  \
-p 10101:10101 \
-p 10050:10050 \
-p 10051:10051 \
-v /var/lib/zabbix/db_data:/var/lib/zabbix/db_data \
--restart unless-stopped \
--init -d zabbix/zabbix-proxy-sqlite3:alpine-7.2.4

docker run --name zabbix-java-gateway-proxy \
--network=zabbix-net \
--restart unless-stopped \
-d zabbix/zabbix-java-gateway:alpine-7.2.4

You can start each of these Zabbix components using the docker run command, however, any change to the container configuration will require you to stop the container, delete it, and execute the docker run command again. You also have another option – you could create a docker compose file and write the necessary configuration in yaml format. When you need to add some changes to the container configuration, run the docker compose down command to remove containers, edit the docker compose file, and run docker compose up command to start them up again with the new configuration:

docker compose -f ./docker_compose_v3_proxy.yaml down
docker compose -f ./docker_compose_v3_proxy.yaml up -d

If you have not mounted volume or directory to container for the data you want to keep, you can copy the data from the container to your host. Otherwise, that data will be gone if you delete the container or use the docker compose down command. So, it is important to set up the persistent storage/volume for the data that needs retaining, so you don’t lose important data from the container when container configuration is changed. You also need to expose the ports for the necessary services for the appropriate components (if they are set up on on separate hosts): zabbix-server, zabbix-proxy, zabbix-agent/zabbix-agent2 (default ports: 10050 for Zabbix agent passive mode, 10051 for Zabbix-agent active mode, some different port for proxy, 10052 for Java gateway).

Here we have the same docker run options written to docker compose file, including the environment variables, mounted directories and exposed ports. You can specify as many services as needed and start them just with docker compose command.

docker_compose_v3_proxy.yaml

services:
  zabbix-proxy-active-01:
    image: "${PROXY_SQLITE3_IMAGE}:${ALPINE_IMAGE_TAG}"
    environment:
      ZBX_HOSTNAME: Zabbix-proxy-active-01
      ZBX_SERVER_HOST: ${ZBX_SERVER_HOST}
      ZBX_PROXYMODE: 0
      ZBX_LISTENPORT: 10101
      ZBX_JAVAGATEWAY_ENABLE: true
      ZBX_JAVAGATEWAY: zabbix-java-gateway-proxy
      ZBX_JAVAGATEWAYPORT: 10052
      ZBX_STARTJAVAPOLLERS: 5
    volumes:
      - /var/lib/zabbix/db_data:/var/lib/zabbix/db_data:rw
    networks:
      - backend
    ports:
      - 10101:10101
      - 10050:10050
      - 10051:10051
    restart: unless-stopped

  zabbix-java-gateway-proxy:
    image: "${JAVA_GW_IMAGE}:${ALPINE_IMAGE_TAG}"
    networks:
      - backend
    restart: unless-stopped

networks:
  backend:
    name: zabbix-net
    external: true

.env

PROXY_SQLITE3_IMAGE=zabbix/zabbix-proxy-sqlite3
JAVA_GW_IMAGE = zabbix/zabbix-java-gateway
ALPINE_IMAGE_TAG=alpine-7.2.4
ZBX_SERVER_HOST=46.101.140.98

You can also use official Zabbix-supplied docker compose files, try them out, and modify them as needed.

You can read more about the official docker compose files here.

Containerized Zabbix components allow us to use test different scenarios within the docker:

Creating HA Zabbix-server nodes
Creating multiple proxies
Creating multiple agents
Adding more Java gateways
Creating multiple frontends
Easily configure Browser monitoring
Configure SNMP traps
Easily make scheduled reports

Deploying multiple redundant Zabbix servers

To enable HA Zabbix server mode, modify both the Zabbix-server container and Zabbix-frontend container configuration environment variables.

For the HA Zabbix server mode, add 2 environment variables:

ZBX_HANODENAME
ZBX_NODEADDRESS

All of the containers are set with the user-defined network, therefore I will use the container name in the ZBX_HANODENAME option instead of the static address, as it will be resolved by docker. If you need to use a different listen port for the trapper, you need to define it using the environment variable ZBX_LISTENPORT. You can omit the port in variable ZBX_HANODENAME, as the ZBX_LISTENPORT (default is 10051) will be applied automatically.

Here is the docker run example for the Zabbix-server HA mode.

docker run --name zabbix-server-mysql-ha1 -t \
-e DB_SERVER_HOST="mysql-server" \
-e MYSQL_DATABASE="zabbix" \
-e MYSQL_USER="zabbix" \
-e MYSQL_PASSWORD="zabbix_pwd" \
-e MYSQL_ROOT_PASSWORD="root_pwd" \
-e ZBX_HANODENAME="zabbix-server-HA1" \
-e ZBX_NODEADDRESS="zabbix-server-mysql-ha1" \
--network=zabbix-net \
-p 10151:10051 \
--restart unless-stopped \
-d zabbix/zabbix-server-mysql:alpine-7.2.4

docker run --name zabbix-server-mysql-ha2 -t \
-e DB_SERVER_HOST="mysql-server" \
-e MYSQL_DATABASE="zabbix" \
-e MYSQL_USER="zabbix" \
-e MYSQL_PASSWORD="zabbix_pwd" \
-e MYSQL_ROOT_PASSWORD="root_pwd" \
-e ZBX_HANODENAME="zabbix-server-HA2" \
-e ZBX_NODEADDRESS="zabbix-server-mysql-ha2" \
--network=zabbix-net \
-p 10251:10051 \
--restart unless-stopped \
-d zabbix/zabbix-server-mysql:alpine-7.2.4

From the frontend container, remove these two environment variables:

ZBX_SERVER_HOST
ZBX_SERVER_PORT

docker run --name zabbix-web-nginx-mysql -t \
-e ZBX_SERVER_HOST="zabbix-server-mysql" \
-e ZBX_SERVER_PORT=10051
-e DB_SERVER_HOST="mysql-server" \
-e MYSQL_DATABASE="zabbix" \
-e MYSQL_USER="zabbix" \
-e MYSQL_PASSWORD="zabbix_pwd" \
-e MYSQL_ROOT_PASSWORD="root_pwd" \
--network=zabbix-net \
-p 80:8080 \
--restart unless-stopped \
-d zabbix/zabbix-web-nginx-mysql:alpine-7.2.4

Once both container configurations are modified, you should be able to see the currently added HA server nodes and their states without issues.

Fig. 1. Containers of HA Zabbix server containers

Fig. 2. Dashboard – system information

You can also execute commands on the container:

# docker exec -it container_name_or_id sh -c "zabbix_server -R ha_status"

Fig. 3. Executing command on container

Containers of HA Zabbix server containers

I’t’s possible to allocate an interactive pseudo-TTY shell, by adding option -ti and specifying shell after the container name or id.

# docker exec -ti container_name_or_id /bin/bash

Fig. 4. Executing command from within container

You can also start multiple proxies at once in docker. This can help to offload preprocessing to the proxy, gather data from the targets behind the firewall, and send collected data back to the Zabbix server, only requiring one port.

Fig. 5.Overall block diagram of Zabbix monitoring opportunities

Deploying multiple Zabbix proxies

First, you must choose the proxy mode and set the environment variable ZBX_PROXYMODE.

For active mode proxy, please define the server host address for a single server or addresses separated by a semicolon in the case of HA Zabbix server configuration (example shown below).

docker run --name zabbix-proxy-active-01 \
-e ZBX_HOSTNAME="Zabbix-proxy-active-01" \
-e ZBX_SERVER_HOST="zabbix-server-mysql-ha1;zabbix-server-mysql-ha2;zabbix-server-mysql-ha3" \
-e ZBX_PROXYMODE="0" \
--network=zabbix-net \
-e ZBX_LISTENPORT=10101  \
-p 10101:10101 \
-v /var/lib/zabbix/db_data:/var/lib/zabbix/db_data \
--restart unless-stopped \
--init -d zabbix/zabbix-proxy-sqlite3:alpine-7.2.4

For passive mode proxy, define the server host address for a single server or addresses separated by a comma in the case of HA Zabbix server configuration (example shown below).

docker run --name zabbix-proxy-passive-01 \
-e ZBX_HOSTNAME="Zabbix-proxy-passive-01" \
-e ZBX_SERVER_HOST="zabbix-server-mysql-ha1,zabbix-server-mysql-ha2,zabbix-server-mysql-ha3" \
-e ZBX_PROXYMODE="1" \
--network=zabbix-net \
-e ZBX_LISTENPORT=10102 \
-p 10102:10102 \
-v /var/lib/zabbix/db_data:/var/lib/zabbix/db_data \
--restart unless-stopped \
--init -d zabbix/zabbix-proxy-sqlite3:alpine-7.2.4

docker_compose_v3_proxies.yaml

services:
  zabbix-proxy-active-01:
    image: "${PROXY_SQLITE3_IMAGE}:${ALPINE_IMAGE_TAG}"
    environment:
      ZBX_HOSTNAME: zabbix-proxy-active-01
      ZBX_SERVER_HOST: zabbix-server-mysql-ha1;zabbix-server-mysql-ha2;zabbix-server-mysql-ha3
      ZBX_PROXYMODE: 0
      ZBX_LISTENPORT: 10101
    volumes:
      - /var/lib/zabbix/db_data:/var/lib/zabbix/db_data:rw
    networks:
      - backend
    ports:
      - 10101:10101
    restart: unless-stopped

  zabbix-proxy-passive-01:
    image: "${PROXY_SQLITE3_IMAGE}:${ALPINE_IMAGE_TAG}"
    environment:
      ZBX_HOSTNAME: zabbix-proxy-passive-01
      ZBX_SERVER_HOST: zabbix-server-mysql-ha1,zabbix-server-mysql-ha2,zabbix-server-mysql-ha3
      ZBX_PROXYMODE: 1
      ZBX_LISTENPORT: 10102
    volumes:
      - /var/lib/zabbix/db_data:/var/lib/zabbix/db_data:rw
    networks:
      - backend
    ports:
      - 10102:10102
    restart: unless-stopped
networks:
  backend:
    name: zabbix-net
    external: true

.env

PROXY_SQLITE3_IMAGE=zabbix/zabbix-proxy-sqlite3
JAVA_GW_IMAGE = zabbix/zabbix-java-gateway
ALPINE_IMAGE_TAG=alpine-7.2.4
ZBX_SERVER_HOST=46.101.140.98

The proxy name in the frontend must be the same as the value set in proxy environment variable ZBX_HOSTNAME! Also, in frontend for active proxies, you don’t need to add the proxy address.

Next, you can set hosts to be monitored by Zabbix-proxies, but make sure to update the agent configuration, so agents accept connections from proxy.

Fig. 6. Hosts monitored by proxy

Fig. 7.List of proxies and hosts monitored by them

Configuring Proxy groups

You can create as many proxy containers as necessary in Docker, and you can also create proxy groups for load balancing (it is based on the number of hosts per proxy).

First, create a proxy group in the frontend:

Set proxy group name
Select failover period
Minimum number of proxies

Fig. 8.Creating a new proxy group

Next, add proxies to the proxy group, and specify the address for active agents and port for the active agents.

Fig. 9. Adding proxy to proxy group

Do not forget to change Zabbix agent configuration for hosts now monitored through the proxy group (add proxy groups IPs/DNS to Server and ServerActive options).

Fig. 10. Creating a new host and monitoring it through proxy group

You can see additional information regarding the proxies in the Frontend section: Administration/ Proxies.

Fig. 11. List of all configured proxies and those belonging to proxy group

Adding more Java gateways

Zabbix server or proxy can communicate with only one Zabbix java gateway, however, you are not limited tin how many Zabbix proxies you create together with Zabbix Java Gateway. You can make an unlimited number of pairs, consisting of Zabbix proxy with Zabbix Java Gateway.

For the containerized Zabbix server, you will need to add these 4 environment variables:

ZBX_JAVAGATEWAY_ENABLE=true
ZBX_JAVAGATEWAY=zabbix-java-gateway-server
ZBX_JAVAGATEWAYPORT=10052
ZBX_STARTJAVAPOLLERS=5

And start the Java gateway for the zabbix-server in docker:

docker run --name zabbix-java-gateway-server -t \
--network=zabbix-net \
--restart unless-stopped \
-d zabbix/zabbix-java-gateway:alpine-7.2.4

Or if you want to add java gateway to the Zabbix proxy, then add these 4 environment variables to Zabbix proxy in docker:

ZBX_JAVAGATEWAY_ENABLE=true
ZBX_JAVAGATEWAY=zabbix-java-gateway-proxy
ZBX_JAVAGATEWAYPORT=10052
ZBX_STARTJAVAPOLLERS=5

And start the java gateway as a container:

docker run --name zabbix-java-gateway-proxy -t \
--network=zabbix-net \
--restart unless-stopped \
-d zabbix/zabbix-java-gateway:alpine-7.2.4

And here we have a host, monitored by zabbix-agent2 through zabbix-proxy-active-02

Fig. 12. Host monitored by proxy with configured Java gateway

Upgrading docker proxies with SQLite3 database

If you have older Zabbix components already running in docker and you have upgraded the server, you will also need to upgrade the proxies.

If you have a container created from the proxy zabbix-proxy-sqlite3 image and want to upgrade it, you will lose the existing data stored in the SQLite3 database. For most users, the database functions as a buffer to temporarily keep the data until it’s sent to Zabbix server and the loss of the proxy database file data is of no consequence.

Once you have updated the image for the container, the proxy will detect the existing old database version on startup. If the directory is mounted to database file, it will delete the database file and create a new one. This will impact those who keep data after sending it to Zabbix server and use the data from the proxy database for other purposes.

Fig. 13. Database upgrade for proxy container with SQLite3 database

Upgrading docker proxies with MySQL database

To upgrade the MySQL database for proxy, log in in the MySQL database, set the log_bin_trust_function_creators flag to 1. Change the proxy image version to a newer one and start the container.

mysql> set global log_bin_trust_function_creators = 1;

If you have not set the flag, you will receive an error of database upgrade.

Fig. 14. Failed database upgrade for proxy with MySQL database

Replace the previous version of the proxy image with the new one, check the log file, and check the docker logs to see when the database schema upgrade has finished. After the upgrade, set the flag back to 0.

mysql> set global log_bin_trust_function_creators = 0;

The upgrade has been successful, and the proxy service has started after that.

Fig. 15. Successful database upgrade for proxy with MySQL database

An official docker image for the proxy with Postgresql database support is not available due to the extensive number of existing images and different versions.

Deploying multiple frontends

You can launch as many frontends as you need if you are experiencing a sudden surge in Zabbix users. Just specify which port to assign for it and you are good to go (don’t forget to also open the port in the firewall).

docker run --name zabbix-web-nginx-mysql1 -t \
-e DB_SERVER_HOST="mysql-server" \
-e MYSQL_DATABASE="zabbix" \
-e MYSQL_USER="zabbix" \
-e MYSQL_PASSWORD="zabbix_pwd" \
-e MYSQL_ROOT_PASSWORD="root_pwd" \
--network=zabbix-net \
-p 80:8080 \
--restart unless-stopped \
-d zabbix/zabbix-web-nginx-mysql:alpine-7.2.4

Fig. 16. One started Zabbix frontend container in docker

docker run --name zabbix-web-nginx-mysql2 -t \
-e DB_SERVER_HOST="mysql-server" \
-e MYSQL_DATABASE="zabbix" \
-e MYSQL_USER="zabbix" \
-e MYSQL_PASSWORD="zabbix_pwd" \
-e MYSQL_ROOT_PASSWORD="root_pwd" \
--network=zabbix-net \
-p 81:8080 \
--restart unless-stopped \
-d zabbix/zabbix-web-nginx-mysql:alpine-7.2.4

Fig. 17. Two started Zabbix frontend containers in docker

docker run --name zabbix-web-nginx-mysql3 -t \
-e DB_SERVER_HOST="mysql-server" \
-e MYSQL_DATABASE="zabbix" \
-e MYSQL_USER="zabbix" \
-e MYSQL_PASSWORD="zabbix_pwd" \
-e MYSQL_ROOT_PASSWORD="root_pwd" \
--network=zabbix-net \
-p 82:8080 \
--restart unless-stopped \
-d zabbix/zabbix-web-nginx-mysql:alpine-7.2.4

Fig. 18. Three started Zabbix frontend containers in docker

Fig. 19. Multiple frontends accessed through different ports

Browser monitoring

Browser monitoring setup has never been easier! Just add two parameters to zabbix-server container config:

ZBX_WEBDRIVERURL=selenium:4444
ZBX_STARTBROWSERPOLLERS=2

And start the web driver in the docker (with a standalone chrome browser):

docker run --name selenium -t\
--network=zabbix-net \
--restart unless-stopped \
-p 4444:4444 \
--shm-size="1g" \
-d selenium/standalone-chrome:latest

Next step: create a new host, add the template, specify which page to monitor with Macro values, and it’s DONE!!!!

Fig. 20. Creating host for monitoring website

Fig. 21. Screenshot of the monitored website

SNMP traps

For the snmptraps to work, the same directory must be shared among the zabbix-server and zabbix-snmptrap container. On the Zabbix-server side, you need to explicitly set snmp environment variable ZBX_ENABLE_SNMP_TRAPS to true and mount directory /var/lib/zabbix/snmptraps.

You also need to add the same volume to the snmptrap container.

And run the snmptraps container (make sure there is no permission issue for the directory)

docker run --name zabbix-snmptraps -t \
-v /var/lib/zabbix/snmptraps:/var/lib/zabbix/snmptraps:rw \
--network=zabbix-net \
-p 162:1162/udp \
--restart unless-stopped \
-d zabbix/zabbix-snmptraps:alpine-7.2-latest

Fig. 22. Received SNMP trap message

Scheduled reports

You can also easily configure scheduled reports by adding 2 additional environment variables to the Zabbix-server. In my case, both of these containers are in the same custom user network, therefore I will use the container name zabbix-web-service in the ZBX_WEBSERVICEURL option.

ZBX_STARTREPORTWRITERS=5
ZBX_WEBSERVICEURL=http://zabbix-web-service:10053/report

Start the Zabbix-web service, specify also these 2 parameters (you can skip those if defaults are used). You can also allow any incoming connections by setting ZBX_ALLOWEDIP=0.0.0.0/0. We discourage this, however.

ZBX_ALLOWEDIP=zabbix-server-mysql
ZBX_LISTENPORT=10053

Before testing scheduled reports, make sure you have enabled and configured the email media type.

Fig. 23. Configured and enabled media type

It is also encouraged to test it and check that you have received the test email.

Fig. 24. Successful media type test response

Fig. 25. Received test response on the selected media type.

Next, configure the user media where the scheduled report will be sent.

Fig. 26.Media type defined for the user

Last, but not least, set the frontend URL in the section Administration/General/Other section. In my case, I set the container name of the frontend and specify the port.

for Apache: http://<server_ip_or_name>/zabbix
for Nginx: http://<server_ip_or_name>

Fig. 27. Configured frontend address for the Frontend URL option

Next, create a scheduled report based on the dashboard of your choice.

Fig. 28.Configuring scheduled report

Check that you have received the test report in your mail.

Fig. 29.Successful scheduled report test.

Fig. 30. Received scheduled report test in the email

Now you know how to set up scheduled reports!

Docker container monitoring

You can also monitor Docker containers with a containerized Zabbix instance*

* Disclaimer: If docker service is not running, Zabbix monitoring will also not function and you will not receive notifications and alerts.

You can also monitor your docker instance with the Zabbix agent 2, however, you will be required to install Zabbix-agent 2 on the host either as a package or build it from the source.

You will also need to give user zabbix access to the docker.sock file. Just add user zabbix to group docker:

# usermod -aG docker zabbix

Otherwise, you will get an error message in items:

Cannot fetch data: Get "http://1.28/info": dial unix /var/run/docker.sock: connect: permission denied.

Go back to the frontend and create a Host for monitoring the docker containers:

Link template: Docker by Zabbix agent 2
Add host to host group
Specify host address or dns name, set the correct connect to option, and specify the agent port (if a default port is used, then set 10050).

Fig. 31. Configuring the host for monitoring the docker container

Now, if some issue happens to other containers, Zabbix will monitor them. But to be notified of an issue, don’t forget to enable and configure the media, user media, media templates, and trigger actions, so that you receive alerts.

Fig. 32.Latest data for the docker host

Thank you for reading – I hope you’ve found this article helpful and informative!

The post Deploying Zabbix Components with Docker and Docker Compose appeared first on Zabbix Blog.

Zabbix server configuration changes

Script configuration in frontend

Manual host script execution

Automating the container crash recovery

Testing container crash automatic recovery

Summary

How does the Zabbix agent 2 do it?

How we will do it

Cloning the template and changing the item type

Cloning the template (again) and making minor modifications

Creating a host and linking the template

Summary

Getting the data to Zabbix from the Docker API

Generating the certificates for the Docker and Zabbix server

Deploying the certificate files and configuring the services

Configuring the monitoring in the Zabbix frontend

The easy way: SSH agent items

The right way: HTTP agent

Extracting the data

Creating the trigger

Getting more data

Testing the trigger

In summary

Tune your database

Monitor the Zabbix database

Monitor the Zabbix server

Check the current state of your Zabbix server

Check the cache utilization

Consequences of running out of configuration cache

Consequences of running out of value and history cache

The result of cache tuning and database tuning

Tune the Zabbix server configuration parameters

Pitfalls of misconfiguration

Summary

Creating a template discovery rule

Creating a template discovery rule: Item prototypes

Creating a template discovery rule: Trigger prototype

Summary

Installing and checking Prometheus Podman exporter

Creating a template and template items

Creating a Discovery rule in template

Creating a template Discovery rule: item prototypes

Creating a template Discovery rule: trigger prototype

Summary

Deploying multiple redundant Zabbix servers

Deploying multiple Zabbix proxies

Configuring Proxy groups

Adding more Java gateways

Upgrading docker proxies with SQLite3 database

Upgrading docker proxies with MySQL database

Deploying multiple frontends

Browser monitoring

SNMP traps

Scheduled reports

Docker container monitoring

The collective thoughts of the interwebz