Tag Archives: json

Handy Tips #39: Extracting metrics from structured data with Zabbix preprocessing

Post Syndicated from Arturs Lontons original https://blog.zabbix.com/handy-tips-39-extracting-metrics-from-structured-data/24163/

Collect structured data in bulk and use Zabbix preprocessing to extract and transform the necessary metrics. 

Collecting data from custom monitoring endpoints such as web applications or custom in-house software can result in the collected data requiring further extraction or transformation to fit our requirements. 

Use Zabbix preprocessing to extract metrics from structured data: 

  • Extract data with JSONPath and XPath expressions
  • Transform XML and CSV data to JSON structures

  • Check for error messages in JSON and XML structures
  • Extract and transform metrics from Prometheus exporter endpoints

Check out the video to learn how to use Zabbix preprocessing to extract metrics from structured data.

Extract metrics from structured data with Zabbix preprocessing: 

  1. Navigate to Configuration → Hosts
  2. Find the host where structured data is collected
  3. Click on the Items button next to the host
  4. Create or open an item collecting structured data
  5. For this example, we will transform CSV to JSON
  6. Open the Preprocessing tab
  7. Select a structured data preprocessing rule
  8. If required, provide the necessary parameters
  9. Optionally, select a validation preprocessing step
  10. For this example, we will check for errors in JSON
  11. Extract a value by using JSONPath or XML XPath preprocessing steps
  12. Press Test to open the test window
  13. Press Get value and test to test the item
  14. Close the test window and press Add or Update to add or update the item
  15. Optionally, create dependent items to extract values from this item

Tips and best practices
  • You check the Handy Tips #37 to learn how to collect structured data from HTTP end-points 

  • For CSV to JSON preprocessing the first parameter allows you specify a CSV delimiter, while the second parameter specifies the quotation symbol 

  • For CSV to JSON preprocessing If the With header row checkbox is marked, the header line values will be interpreted as column names 

  • For details on XML to JSON preprocessing, refer to our serialization rules for more details. 

Learn how to leverage the many types of data collection provided by Zabbix and empower your data collection and processing. Sign up for our Zabbix Certified Specialist course, where under the guidance of a Zabbix certified trainer you will learn more about different types and technologies of monitoring and learn how to get the most out of your Zabbix instance. 

The post Handy Tips #39: Extracting metrics from structured data with Zabbix preprocessing appeared first on Zabbix Blog.

JSON is your friend – Certificate monitoring on Microsoft CA server

Post Syndicated from Tibor Volanszki original https://blog.zabbix.com/json-is-your-friend-certificate-monitoring-on-microsoft-ca-server/20697/

Introduction

By transforming our data into JSON we can achieve great results with Zabbix without the need to have a complex external script logic. The article will provide an example of obtaining a set of master data with a single PowerShell script and then using the Zabbix native functionality to configure low-level discovery and collect the required metrics from the master JSON data set.

In this example, we will implement certificate monitoring by using a Microsoft Windows Certificate Authority server. The goal is to see how many days we have before our internally signed certificates will expire. As an extra, we will be able to filter the monitored items by requestors and template names.

  • Target system version: MS Windows Server 2019 (also tested on 2012)
  • Zabbix server & agent version: 6.0.3 (Other Zabbix versions should be supported too with no or minimal changes)

Core logic

The below diagram shows the concept itself and you will find a detailed guide to implement this step by step in the next sections.

Core workflow

 

Sample output of Latest Data:

Prerequisites

To start, you will need to deploy Zabbix agent 2 on your target system – confirm that the Zabbix agent can communicate with your Zabbix server.  For the calculated item, we will require local time monitoring. The easiest way is to use the key “system.localtime”, which is already provided by the default Windows OS monitoring template. This item is intentionally not included within the certificate monitoring template to avoid key conflicts.

Below you can find the Powershell script which you have to implement: (name: “certmon_get_certs.ps1″) :

To import module PSPKI you may have to install the module first: https://pkisolutions.com/tools/pspki/

Import-Module PSPKI
$ca_hostname=$args[0]
$start = (Get-Date).AddDays(-7)
Get-IssuedRequest -CertificationAuthority $ca_hostname -Filter "NotAfter -ge $start" | Select-Object -Property RequestID,Request.RequesterName,CommonName,NotAfter,CertificateTemplateOid | ConvertTo-Json -Compress

*for Windows 2012 systems use CertificateTemplate instead of CertificateTemplateOid within this script.

This script expects one parameter, which is your CA server’s FQDN. That name will be loaded into variable “$ca_hostname”, so for testing purposes, you can just replace it with a static entry. Based on your needs you can adjust the “AddDays” parameter. Essentially this means to track back certificates, which are already expired for up to 7 days. Consider a situation, when you miss one just before a long weekend.

Script output

Let’s check the first part of the main command without further piping:

Simple script output without additional properties

As you can see, we are getting some basic information about the certificate. We do not need all the returned lines for the next steps, so it is better to filter the output down. Before we do that, we will use the option called “-Property”, and if you use it with a wildcard character, then it will list all the available parameters for a certificate. If you need more than the basic output, then you can list the extra parameters by using this option. Please be aware, that this will add extra lines compared to the basic output, but it will not do any filtering (the common lines will always remain visible).

Output with “-Property *”

Compare this with the output after using “Select-Object -Property RequestID,Request.RequesterName,CommonName,NotAfter,CertificateTemplateOid”

Output after using property filters

This looks good for us, but it is still not machine-readable. Let’s add the JSON output conversion, but without the “-Compress” option first:

JSON converted compressed output

What is especially great in this conversion, is that the “CertificateTemplateOid” part got 2 child entries, so later we can target the “FriendlyName” entry for discovery. Lastly, adding the “-Compress” option will help us to use less space by removing the white spaces and newlines from the output.

Depending on the amount of issued and valid certificates the output can be huge, especially without any filtering and compression (even megabytes in size). The current Zabbix server version (6.0.3) supports only 512KB as item output, this is why the output reduction is crucial. In my example the text data of one certificate takes approx 300 bytes, so 512KB will result in a limit of 1747 certificates. In case you are expecting more than this amount of ACTIVE certs within your CA, then I recommend cloning the PS script and adding some extra filtering to each variant (filter for template name / requestor / OU) and adjusting the template accordingly. Another approach would be to monitor the certificates, which will expire in the coming N days in case you have too many entries.

Agent configuration

To run the defined script, you have to allow it within the Zabbix Agent configuration file. You can either modify the main config or just define the extra lines within an additional file under zabbix_agent2.d folder. Below you can find the additional Zabbix agent configuration lines:

AllowKey=system.run[powershell -NoProfile -ExecutionPolicy bypass -File "C:\Program Files\Zabbix Agent 2\scripts\certmon_get_certs.ps1" *]
Timeout=20

The wildcard character at the end is needed to specify any hostname, which is expected by the script. The default timeout is 3 seconds, which is unfortunately insufficient. Importing Module PSPKI alone takes a few seconds, so the overall execution time is somewhere between 5 and 10 seconds. My assumption is that more certificates will not increase this significantly, but some extra seconds can be expected. 20 seconds sounds like a safe bet.

We are done with the pre-requisites, now we can start the real work!

Template

Let’s create our template from scratch.

  • Name: “Microsoft Certificate Authority – Certificate monitoring”
  • Host group: any group will do
Master item

We need only the following item:

  • Name: “Get certificate data”
  • Type: “Zabbix agent / Zabbix agent (active)” – for testing I recommend the passive mode
  • Key: system.run[powershell -NoProfile -ExecutionPolicy bypass -File “C:\Program Files\Zabbix Agent 2\scripts\certmon_get_certs.ps1″ {HOST.DNS}]”
  • Type of information: Text”
  • Update interval: 6h”
  • History: 1d”
Item configuration example

Assign the template to a CA server and you can test the item already. The result should be a big block of data in JSON format. To review the output I recommend the following external websites:

The latter one is especially helpful to find the correct JSON path, which we will require in the upcoming steps.

Item output sample
Measuring the script execution time

To measure the execution time, you can test it by using Zabbix get, which can connect to your Zabbix agent and request the item value over a CLI:

time zabbix_get -s [CA_SERVER_FQDN] --tls-connect psk --tls-psk-identity [PSK_IDEN] --tls-psk-file [PSK_FILE] -k 'system.run[powershell -NoProfile -ExecutionPolicy bypass -File "C:\Program Files\Zabbix Agent 2\scripts\certmon_get_certs.ps1" [CA_SERVER_FQDN]]'

This will give you the normal output and the execution time info:

real   0m5.771s
user   0m0.003s
sys    0m0.005s

To test the size of the output, redirect your command output to any file and measure it by “du -sk” to get it in kilobytes.

zabbix_get -s [CA_SERVER_FQDN] --tls-connect psk --tls-psk-identity [PSK_IDEN] --tls-psk-file [PSK_FILE] -k 'system.run[powershell -NoProfile -ExecutionPolicy bypass -File "C:\Program Files\Zabbix Agent 2\scripts\certmon_get_certs.ps1" [CA_SERVER_FQDN]]' > output.test
du -sk output.test
Low-level discovery rule definition

If the above part works just fine, then proceed with defining the low-level discovery rule as per the below example:

  • Name: Certificate discovery”
  • Type: Dependent item”
  • Key: certificate.discovery”
  • Master item: select our previously created item
  • Keep lost resources period: “6h”
Discovery rule configuration example

Then switch to LLD macros and define the below lines:

  • “{#COMMON_NAME}”: $.CommonName”
  • “{#REQUESTOR_NAME}”: $.[“Request.RequesterName”]”
  • “{#REQUEST_ID}”: $.RequestID”
  • “{#TEMPLATE_NAME1}”: “$.CertificateTemplate”
  • “{#TEMPLATE_NAME2}”: $.CertificateTemplateOid.FriendlyName”
Discovery rule LLD macro definitions

Some explanation:
The first LLD macro is self-explanatory – it obtains the certificate’s common name. The second one is also trivial, except the special marking, which is required due to the dot character in the middle. The third one is also simple, but the last 2 lines are somewhat special. If you have a fresh OS version, then most probably you will need only the 5th line without the 4th. In case you have a Windows server 2012 system, then you will need only the 4th line without the 5th. Why? Because of Windows 🙂 For testing you can keep both and then later remove the unnecessary one as well as the number suffix.

Now you are ready to create your item prototypes and this is where the real magic starts.

Certificate expiration date item prototype – Dependent item

Define the new item prototype as follows:

  • Name: Certificate [ ID #{#REQUEST_ID} ] {#COMMON_NAME} – Expiration date”
  • Type: Dependent item”
  • Key: certificate.expiration_date[{#REQUEST_ID}]”
  • Type of information: “Numeric (unsigned)”
  • Master item: pick the previously created master item
  • Units: unixtime”
  • History: 1d”
Item prototype example

Tags

  • cert_requestor“:  “{{#REQUESTOR_NAME}.regsub(“\\(\w+)”, “\1″)}”
  • cert_template1“: {#TEMPLATE_NAME1}”
  • cert_template2“: “{#TEMPLATE_NAME2}”
  • “scope”: certifcate / expiration date”
Item prototype tag definitions

As mentioned previously, it makes sense to keep only one template tag later, but for now, such an approach is fine.

The first line requires some explanation:
The requestor name starts with a domain prefix followed by 2 backslashes. If you are submitting CSRs from different domains to this CA server, then you can remove the extra formatting, but in a simple setup, we do not need the domain prefix, since it will be the same for all requestors.
Example: “DOMAIN\\someuser → someuser”

Preprocessing

  • JSONPath: $[?(@.RequestID == {#REQUEST_ID})].NotAfter”
  • Regular expression: (\d+) \1″
  • Custom multiplier: 0.001″
  • Discard unchanged with heartbeat: 1d”
Item prototype preprocessing step definitions

Explanation:
Since this item is a dependent item, it will point back to our master item, which returns a data block in JSON. Due to the nature of the discovery definitions, we are running a while loop, which is already loaded with our variables (the LLD macros). Therefore the “{#REQUEST_ID}” already has a numerical value within each cycle. With this number, we can go back to the original item and target that exact certificate, which has the same ID. Then we are interested in the NotAfter value considering the selected certificate.

You can find many other examples within Zabbix documentation: jsonpath functionality
At this point, we have the extracted value of the expiration date, but it is quite raw at the moment:

\/Date(1673594922000)\/

In the next step, we are taking the numerical part and then we have to apply a multiplier of 0.001 since by default the time is given in milliseconds. After this, we have an integer, which can be converted to a human-readable form by using the unit unixtime”. The last line is just a standard discard unchanged entry.

Since our discovery object is also a dependent item, you have to execute our master item to run the low-level discovery rule. The first execution will result in the creation of your certificate items and only the second execution of the master item will execute them all at once. After this point, you should have N certificate objects created and each should have a valid expiration date. This is already something, for which you could define a trigger, but personally, I prefer to see the remaining days and not the exact date itself.

Days to expire item prototype – Calculated item

Let’s define yet another item prototype as follows:

  • Name: Certificate [ ID #{#REQUEST_ID} ] {#COMMON_NAME} – Days to expire”
  • Type: Calculated”
  • Key: certificate.remaining_days[{#REQUEST_ID}]”
  • Type of information: Numeric (float)”
  • Formula: (last(//certificate.expiration_date[{#REQUEST_ID}])-last(//system.localtime))/86400″
  • Update interval: 6h”
  • History: 1d”
Remaining days to expiration item prototype example

Please do not forget, that you require an existing local time item, which is not provided by this template (but available within Windows by Zabbix agent*” template).

Tags:

Copy the same tags from the first prototype and only change the last tag to scope: certificate / remaining days

Remaining days to expiration item prototype tag definitions

Preprocessing:

  • Regular expression: ^(-?\d+)”: “\1”
  • Discard unchanged with heartbeat: 1d”
Remaining days to expiration item prototype preprocessing steps

As a result, this will give you a simple number with the remaining days to the expiration date. Then you can decide which item to use in the trigger to implement proper alerting based on your needs.

Certificate expiration trigger prototype

In my case I am just using a simple trigger expression for the remaining days:

  • Name:Certificate will expire within 30 days – {#COMMON_NAME}”
  • Operational data:Expires in {ITEM.LASTVALUE1} days”
  • Severity: up to you
  • Expression:last(/Microsoft Certificate Authority – Certificate monitoring/certificate.remaining_days[{#REQUEST_ID}])<=30″
Trigger prototype example

Tags:

  • cert_cn: “{#COMMON_NAME}”
  • cert_id“:{#REQUEST_ID}”
Trigger prototype tag definitions

When you check the relevant certificates in the Latest data section, then you can do the filtering by the item-based tags. Since we are adding the cert CN and ID only to the trigger, these will appear only in case of alerts. Based on your needs you can implement additional tags, you just have to adjust the PS script to show more properties. When you extend the input data, please always consider the 512KB limit or the configured timeout.

The logic defined in this example can be applied to any JSON formatted data.

Enjoy!

The post JSON is your friend – Certificate monitoring on Microsoft CA server appeared first on Zabbix Blog.

Advanced Zabbix API – 5 API use cases to improve your API workfows

Post Syndicated from Arturs Lontons original https://blog.zabbix.com/advanced-zabbix-api-5-api-use-cases-to-improve-your-api-workfows/16801/

As your monitoring infrastructures evolve, you might hit a point when there’s no avoiding using the Zabbix API. The Zabbix API can be used to automate a particular part of your day-to-day workflow, troubleshoot your monitoring or to simply analyze or get statistics about a specific set of entities.

In this blog post, we will take a look at some of the more advanced API methods and specific method parameters and learn how they can be used to improve your API workflows.

1. Count entities with CountOutput

Let’s start with gathering some statistics. Let’s say you have to count the number of some matching entities – here we can use the CountOutput parameter. For a more advanced use case – what if we have to count the number of events for some time period? Let’s combine countOutput with time_from and time_till (in unixtime) and get the number of events created for the month of November. Let’s get all of the events for the month of November that have the Disaster severity:

{
"jsonrpc": "2.0",
"method": "event.get",
"params": {
"output": "extend",
"time_from": "1635717600",
"time_till": "1638223200",
"severities": "5",
"countOutput": "true"
},
"auth": "xxxxxx",
"id": 1
}

2. Use API to perform Configuration export/import

Next, let’s take a look at how we can use the configuration.export method to export one of our templates in yaml:

{
"jsonrpc": "2.0",
"method": "configuration.export",
"params": {
"options": {
"templates": [
"10001"
]
},
"format": "yaml"
},
"auth": "xxxxxx",
"id": 1
}

Now let’s copy and paste the result of the export and import the template into another environment. It’s extremely important to remember that for this method to work exactly as we intend to, we need to include the parameters that specify the behavior of particular entities contained in the configuration string, such as items/value maps/templates, etc. For example, if I exclude the templates parameter here, no templates will be imported.

{
"jsonrpc": "2.0",
"method": "configuration.import",
"params": {
"format": "yaml",
"rules": {
"valueMaps": {
"createMissing": true,
"updateExisting": true
},
"items": {
"createMissing": true,
"updateExisting": true,
"deleteMissing": true
},
"templates": {
"createMissing": true,
"updateExisting": true
},

"templateLinkage": {
"createMissing": true
}
},
"source": "zabbix_export:\n version: '5.4'\n date: '2021-11-13T09:31:29Z'\n groups:\n -\n uuid: 846977d1dfed4968bc5f8bdb363285bc\n name: 'Templates/Operating systems'\n templates:\n -\n uuid: e2307c94f1744af7a8f1f458a67af424\n template: 'Linux by Zabbix agent active'\n name: 'Linux by Zabbix agent active'\n 
...
},
"auth": "xxxxxx",
"id": 1
}

3. Expand trigger functions and macros with expand parameters

Using trigger.get to obtain information about a particular set of triggers is a relatively common practice. One particular caveat that we have to consider is that by default macros in trigger name, expression or descriptions are not expanded. To expand the available macros we need to use the expand parameters:

{
"jsonrpc": "2.0",
"method": "trigger.get",
"params": {
"triggerids": "18135",
"output": "extend",
"expandExpression":"1",
"selectFunctions": "extend"
},
"auth": "xxxxxx",
"id": 1
}

4. Obtaining additional LLD information for a discovered item

If we wish to display additional LLD information for a discovered entity, in this case – an item, we can use the selectDiscoveryRule and selectItemDiscovery parameters.
While selectDiscoveryRule will provide the ID of the LLD rule that created the item, selectItemDiscovery can point us at the parent item prototype id from which the item was created, last discovery time, item prototype key, and more.

The example below will return the item details and will also provide the LLD rule and Item prototype IDs, the time when the lost item will be deleted and the last time the item was discovered:

{
"jsonrpc": "2.0",
"method": "item.get",
"params": {
"itemids":"36717",
"selectDiscoveryRule":"1",
"selectItemDiscovery":["lastcheck","ts_delete","parent_itemid"]
}, "auth":"xxxxxx",
"id": 1
}

5. Searching through the matched entities with search parameters

Zabbix API provides a couple of standard parameters for performing a search. With search parameter, we can search string or text fields and try to find objects based on a single or multiple entries. searchByAny parameter is capable of extending the search – if you set this as true, we will search by ANY of the criteria in the search array, instead of trying to find an entity that matches ALL of them (default behavior).

The following API call will find items that match agent and Zabbix keys on a particular template:

{
"jsonrpc": "2.0",
"method": "item.get",
"params": {
"output": "extend",
"templateids": "10001",
"search": {
"key_": ["agent.","zabbix"]
},
"searchByAny":"true",
"sortfield": "name"
},
"auth": "xxxxxx",
"id": 1
}

Feel free to take the above examples, change them around so they fit your use case and you should be able to quite easily implement them in your environment. There are many other use cases that we might potentially cover down the line – if you have a specific API use case that you wish for us to cover, feel free to leave a comment under this post and we just might cover it in one of the upcoming blog posts!