Post Syndicated from Janis Eidaks original https://blog.zabbix.com/zabbix-and-the-docker-api-part-3-control/32961/

In this blog post, you will learn how to add a simple container remote control capability to Zabbix in order to start, stop, or restart containers from within the discovered host.

You might be wondering, why spend the effort to create a host for each template? Well, that’s because we define a manual script to control the container from within the Zabbix frontend. That’s neat, right?  And why stop there? We can also implement a trigger action that automatically restarts the container if it crashes for any reason.

Zabbix server configuration changes

First, we will require global script execution in your Zabbix server configuration:

# nano /etc/zabbix/zabbix_server.conf
EnableGlobalScripts=1
# systemctl restart zabbix-server

Script configuration in frontend

We can create a script in the section Alerts > Scripts. In the script, fill out the specified parameters shown below – the scope, type, and command. Then specify to which hosts this command will apply, as well as the user group that will be able to execute this. This script will take advantage of the user macros and built-in macros to fill the required info in the command to make a correct post request.

● Script
▪ Name: Container action
▪ Scope : Manual host action
▪ Type: Script
▪ Execute on: Zabbix server
▪ Commands: curl -sS -X POST https://{$DOCKER.IP}:{$DOCKER.PORT}/containers{HOST.NAME}/{MANUALINPUT} --cert /etc/zabbix/ssl/certs/client-cert.pem --key /etc/zabbix/ssl/keys/client-key.pem --cacert /etc/zabbix/ssl/ca/ca.pem
▪ Description: Manual action to restart,stop,start container
▪ Host group: Selected: Docker
▪ User group: Zabbix administrators 
▪ Req host perm: Write

● Advanced configuration
▪ Enable user input Check
▪ Input prompt Specify action for container {HOST.NAME}:
▪ Input type: Dropdown 
▪ Dropdown options: restart,stop,start
▪ Enable Confirmation: Check
▪ Confirmation text: Confirm to {MANUALINPUT} container: {HOST.NAME}

Manual host script execution

We can go the Menu section Monitoring > Hosts, select the host, and click on it. In the menu, you will have an additional script available for the Docker hosts group Container action. This manual action is also available in some other frontend sections.

Once you click on the Container action, you will have several options available. You can start, restart, or stop the container.

You will have a confirmation window asking if this is the right action you want to perform.

The status can also be checked in the host’s latest data menu, once the metric is collected (1 minute for the master item). The item Container /zabbix-agent2: Running shows that this container is not running, and another item displays the exit code 0, which means the process stopped normally with no issue whatsoever.

Some items report the status in numerical format, e.g., 0 , 1 , 2, and so on. To make it human-readable, we use value maps, which display the value in a meaningful, human-friendly way. The screenshot below shows the value map for container health status. So, instead of looking at value 3 for container health (which is meaningless for us and will require reading the documentation) we will be shown value healthy (3).

Automating the container crash recovery

What if your container crashes for some reason? Well, you will get a problem event, which you can use to receive notifications about issues with containers. You can also automate the container recovery process. For example, create a trigger action that will restart the container 3 times with an interval of 2 minutes.

If it does not resolve the issue, only then send a message to the admin. There is no reason to repeatedly restart the service until the end of time – if a few attempts did not work, most likely it will require human intervention to solve the issue.

So here are the script parameters for the action operation:

● Script
▪ Name: Restart container
▪ Scope: Action operation
▪ Type: Script
▪ Execute on: Zabbix server
▪ Commands: curl -sS -X POST https://{$DOCKER.IP}:{$DOCKER.PORT}/containers{HOST.NAME}/restart --cert /etc/zabbix/ssl/certs/client-cert.pem --key /etc/zabbix/ssl/keys/client-key.pem --cacert /etc/zabbix/ssl/ca/ca.pem
▪ Description: Restart container
▪ Host group: Selected: Docker

Now we have to define a trigger action in order to make use of this script and send a notification to admin if that fails.

Let’s create a new trigger action:

● Action tab
▪ Name: Automatic container restart
▪ Type of calc: And (A and B)
▪ condition: Host group equals Docker
▪ condition: Event name contains Container has been stopped with error code
▪ Enabled: Check

● Operations tab
▪ Default operation step duration: 2m

Add operation
▪ Operation: Current host: Check
▪ Steps: 1 -3
Add operation
▪ Operation: Send message
▪ Steps: 4 – 4
▪ Custom message: Check
▪ Subject: Automated restart failed to bring container up: {HOST.NAME}
▪ Message: <b>Host: {HOST.NAME}<br>
           <b>Problem started at {EVENT.TIME} on {EVENT.DATE}<br>
           <b>Operational data: {EVENT.OPDATA}<br>
           <b>Original problem ID: {EVENT.ID}<br>

The action should look like the screenshot below. Save it.

Testing container crash automatic recovery

I will stop the container with the command docker kill zabbix-agent2. The container has been stopped with an exit code different from 0, so when the item receives the data (in my case, after 1 minute) I get a problem event. The trigger action executes the script Container restart immediately, after 2 minutes, and after 4 minutes if the problem event has not been resolved.

This script successfully restarted the container. The container is running again, and the problem event is resolved.

Let’s see if I am quick enough to kill the west proxy container repeatedly, before the item collects the data with the container running state. Well, I managed to be faster, so now you will see what happens when it “fails” to bring the container back to running state. Here in the Actions, we can see that Zabbix executed the script three times (I also stopped the container 3 times fast enough!) after which the action sent a notification to the admin about a failure to bring the container up with restarts.

I have received the message that the container restart was unable to bring the container to a running state and requires human interaction to fix this.

Summary

Now you know how to plan ahead and make use of the built-in capabilities of Zabbix to solve the issue without human intervention (where possible) and only get notifications when the automatic remediation attempt fails.

The post Zabbix and the Docker API, Part 3: Control appeared first on Zabbix Blog.

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/zabbix-and-the-docker-api-part-2-adapt/32912/

In this blog post, I will show you how to create a template for monitoring your Docker server with only API calls (without the Zabbix agent 2). Instead of creating a template, templated items, LLD rules, and trigger prototypes from scratch, we will adapt them from the existing template “Docker by Zabbix agent 2.”

How does the Zabbix agent 2 do it?

If you are wondering how the Zabbix agent 2 collects the data, you can look into the source code and see the magic behind the scenes: https://github.com/zabbix/zabbix/blob/master/src/go/plugins/docker/metrics.go.

In short, it uses a Unix non-TCP socket, makes the Docker API requests on the host, and returns JSON objects. Hey, we already know how to use it ourselves from the previous blog post, right?

For the Zabbix agent 2 to work with the Docker template, it needs access to the Unix socket, either by adding the user Zabbix to the group: Docker or running the Zabbix-agent2 as root.

How we will do it

I can adapt this template, improvise whenever I encounter a non-existing metric, and overcome any technical challenge with effort! For the most part, in the template we have a few Zabbix agent items that collect data in bulk and a lot of dependent items (and dependent item prototypes from the LLD rules).

The path forward is quite straightforward – we will clone the template and replace the Zabbix agent item type with the HTTP agent type item and add additional parameters shown below. I will also add additional user macros on the template, including the Docker server IP address, port, CA, SSL certificate, and key file names so that these can be adjusted on the host level.

Cloning the template and changing the item type

First, clone the template “Docker by Zabbix agent 2” and give it a new name: “Docker stats by HTTP.” Next, modify the Templated Zabbix agent type item configuration with the following parameters:

Modify “Docker by HTTP” template items:

● Modify item: Get containers
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/containers/json?all=true     
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● Modify item: Get data_usage
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/system/df
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● Modify item: Get images
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/images/json
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

●Modify item: Get info
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/info
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● Modify item: Ping
  ▪ Type HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/_ping
  ▪ Type of inf: Text
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}
♯ Preprocessing (additional first step)
  ▪ Boolean to Decimals

We also need to modify the LLD rule configuration. Change item type from Zabbix agent type to HTTP agent type:

Modify LLD rule: Containers discovery
● Discovery rule
  ▪ Type: HTTP agent
  ▪ Key: docker.containers.discovery[true]
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/containers/json?all=true     
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}
  ▪ Update interval: 1h
● LLD macros
  ▪ {#ID}   $.Id
  ▪ {#NAME} $.Names.first()

Modify LLD rule: Images discovery
● Discovery rule
  ▪ Type: Dependent item
  ▪ Master item: item> Get images
● LLD macros
  ▪ {#ID}   $.Id
  ▪ {#NAME} $.RepoTags

After that, we will also make changes in the LLD rule “Containers discovery” by modifying a few existing item prototypes (Zabbix agent type) and adding a new item. Below are the item prototypes that require modification:

● In LLD rule Containers discovery, modify parameters for item prototype: Container {#NAME}: Get info
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/containers{#NAME}/json       
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● In LLD rule Containers discovery, modify item prototype: Container {#NAME}: Get stats
  ▪ Type: HTTP agent
  ▪ URL: https://{$DOCKER.IP}:{$DOCKER.PORT}/containers{#NAME}/stats?stream=false  
  ▪ SSL verify peer: Checked
  ▪ SSL verify host: Checked
  ▪ SSL certificate file: {$SSL.CERTIFICATE.FILE}
  ▪ SSL key file: {$SSL.KEY.FILE}
  ▪ SSL key password: {$SSL.KEY.PASSWORD}

● In LLD rule Containers discovery, modify parameters for item prototype: Container {#NAME}: CPU percent usage
  ▪ Type: Calculated
  ▪ Formula: last(//docker.container_stats.cpu_usage.total.rate["{#NAME}"])/last(//docker.container_stats.system_cpu_usage.total.rate["{#NAME}"])*last(//docker.container_stats.online_cpus["{#NAME}"])*100
♯ Preprocessing (delete preprocessing step JSONPath)

● In LLD rule Containers discovery, add new item prototype: Container {#NAME}: System CPU total usage per second
  ▪ Name: Container {#NAME}: System CPU total usage per second
  ▪ Type: Dependent item
  ▪ Key: docker.container_stats.system_cpu_usage.total.rate["{#NAME}"]
  ▪ Type of information Numeric (float)
  ▪ Master item    prototype > Container {#NAME}: Get stats

♦ Tags (name:value)        
  ▪ component:cpu
  ▪ container:{#NAME}      

♯ Preprocessing

  ▪ JSONPath       $.cpu_stats.system_cpu_usage
  ▪ Change per second
  ▪ Custom multiplier: 1.0E-9

Cloning the template (again) and making minor modifications

Next, I will clone the template “Docker statistics by HTTP” and give the copy a new name  – “Docker containers by HTTP.” In the template “Docker containers by HTTP,” delete the LLD rule Images discovery; delete templated items; from the LLD rule “Containers discovery” rule, delete all prototype entities (item prototypes), and add a filter in the LLD rule (shown below):

In LLD rule Containers discovery rule, in Filter tab: add additional filter option
Filters [type of calculation: A and B and C]
   ▪ {#ID} matches {HOST.HOST}

I have also created a host group “Docker” where the discovered container hosts will be added. In the template “Docker statistics by HTTP” delete all item prototypes in the LLD rule “Containers discovery.” We will create a Host prototype in the LLD discovery rule “Containers discovery” – the parameters are shown below:

Host prototype in LLD rule: Containers discovery
  ▪ Host name:     {#ID}
  ▪ Visible name:  {#NAME}
  ▪ Templates:     Docker containers by HTTP

Creating a host and linking the template

Now all that is left is to create a host and link a template: “Docker statistics by HTTP.” Do not forget to add the correct Docker IP address or DNS name in the user macro.

I have created a new host “Docker server,” linked a template, and modified the user macro for the Docker IP address. This host will collect Docker overall statistics. After the LLD discovery execution, the container names will be automatically discovered and container hosts will be created with a linked template.

If for some reason you are monitoring multiple Docker instances, you could have the same container names discovered, which will lead to LLD errors, as there can’t be hosts with the same name (container ID) or visible name (container name). Quick solution – for each Docker instance, add a prefix to each container name. Another solution – don’t split the template into two parts, then the items will be discovered under the same host, and you will not have this issue.

The Docker server host shows general information about the Docker server’s overall state and status:

A host will be created automatically for each discovered container and will collect the container-specific performance metrics:

Summary

Now you and I know a little bit more about how Zabbix agent2 is collecting Docker metrics. This blog post has shown you how to improvise and adapt existing templates with different data collection methods. Zabbix is a very versatile tool that you can use in multiple ways to get the data if you have some technical constraints. The included template can also be used as is, or you can modify it to suit your needs.

The post Zabbix and the Docker API, Part 2: Adapt appeared first on Zabbix Blog.