Tag Archives: monitoring

Automate SageMaker HyperPod incident triage and root-cause-analysis with AWS DevOps Agent

2026-08-10 Tomonori Shimomura

Post Syndicated from Tomonori Shimomura original https://aws.amazon.com/blogs/devops/automate-sagemaker-hyperpod-incident-triage-and-root-cause-analysis-with-aws-devops-agent/

Introduction

Large-scale machine learning workloads: training, fine-tuning, and inference run on clusters of hundreds to thousands of GPU instances for days or weeks at a stretch. Keeping operational visibility across a fleet of this size is a constant challenge: hardware health events, node lifecycle transitions, capacity fluctuations, and workload-level issues appear in the event stream around the clock, including nights and weekends.

Amazon SageMaker HyperPod is a purpose-built managed cluster service that lets you run distributed model training, fine-tuning, and inference across hundreds of accelerated instances. It provides built-in resiliency that automatically detects and replaces faulty hardware, so long-running jobs can continue with minimal interruption.

For teams operating these clusters, the scale still creates a fundamental tension: you need continuous visibility into your fleet, but you can’t afford to keep engineers watching the event stream 24/7.

What HyperPod resiliency already handles

SageMaker HyperPod’s built-in resiliency layer automatically detects and self-heals instance-level GPU failures. When the Health Monitoring Agent (HMA) identifies a bad GPU, the HyperPod resiliency layer drains, reboots, or replaces the node depending on the error type, and the job resumes without human intervention. This is exactly what you want: routine hardware failures are handled automatically so your training runs keep going.

This solution does not replace HMA or any part of HyperPod’s resiliency. It adds an autonomous investigation layer on top, using the cluster events and health signals that HMA and HyperPod already produce as its input.

Operational conditions where a human still wants to be in the loop

With that self-healing in place, there are operational conditions where a human still wants to be in the loop or decide:

Configuration issues: a lifecycle-script change you made, a misconfigured mount, or a networking/security change that causes provisioning failures on every new node.

Capacity conditions: a replacement waiting on capacity in the pool, where the operator needs to know recovery is in flight and can decide whether to intervene.

Recurring hardware faults: each fault self-heals correctly, but the same GPU error signature recurring across three or more replacements on one instance group in a week is a pattern worth surfacing to an operator as a single signal.

Workload-level conditions: Pods stuck in CrashLoopBackOff for hours, nodes sitting NotReady, or GPU allocation chronically low.

Without automation, these conditions push operators into round-the-clock manual triage: correlating events across the SageMaker control plane, Amazon EKS, and Amazon CloudWatch, and deciding whether HyperPod is still recovering or needs a hand.

Opportunity: AWS DevOps Agent as a 24/7 companion

AWS DevOps Agent provides an autonomous incident-response platform that can be taught a domain’s operational model through custom skills. By wiring your HyperPod cluster into DevOps Agent, you get a 24/7 companion that complements HyperPod’s self-healing. It watches for the operational conditions that still need a human decision, triaging them, root-causing them, and delivering a clear verdict with recommended actions.

By design, DevOps Agent is configured to run in observe-and-report mode for this integration – it is not granted SSM, SSH, or action-taking permissions against your cluster or its nodes. The agent reads cluster events, control-plane state, Kubernetes objects, and CloudWatch logs to reconstruct what happened; every corrective action (node reboots, replacements, drains) continues to be performed by HyperPod’s own resiliency layer or by an operator responding to the emailed verdict. This read-only boundary is deliberate: it keeps the agent’s blast radius zero while still delivering the correlation and triage value.

In this post, you will learn how to connect any SageMaker HyperPod cluster (either the EKS or Slurm Orchestrator option) to AWS DevOps Agent. Conditions are auto-detected, triaged, root-caused from cluster state and CloudWatch logs, and emailed as a clear verdict. You will also see how the solution can be extended to detect additional conditions specific to your workloads.

Solution Overview

What this solution delivers

This solution wires any SageMaker HyperPod cluster into AWS DevOps Agent so that operational conditions calling for a human decision are auto-detected, triaged, root-caused, and delivered as a human-readable verdict email. Specifically, you get:

Autodetection of HyperPod conditions that complement resiliency self-healing, from the live SageMaker event stream and a periodic Kubernetes-state audit.

Triage + root-cause analysis by the DevOps Agent, taught HyperPod’s operational model via two custom skills. It reconstructs the incident timeline and decides whether HyperPod is still recovering or needs an operator.

Human-readable verdict emails: Monitor (recovery in flight, here’s the ETA), Escalate (you need to act, here’s why and what to do), or Resolved (auto-recovery closed the loop). Noise is filtered out.

Extensibility: customize what conditions are detected (by modifying the periodic-audit Lambda) and how the agent reasons about them (by editing the plain-English skills).

The following screenshot shows the DevOps Agent incident response dashboard with example verdict emails for three common fault types:

DevOps Agent incident response dashboard showing investigation list and timeline, with three email verdict examples for GPU NVLink fault, lifecycle-script bootstrap failure, and insufficient-capacity errors

Architecture

The whole solution deploys one AWS CloudFormation stack per cluster. Two event paths feed the DevOps Agent, and one path carries its verdicts back out to you.

Architecture diagram showing the event flow from HyperPod Health Monitoring Agent through EventBridge to DevOps Agent and email notification

This architecture shows a 1:1 relationship between a HyperPod cluster and a DevOps Agent space, and the deployment instructions in this post follow that model. If you need to associate multiple clusters with a single Agent Space, you can customize the CloudFormation template and the ClusterFilter parameter to widen the allowlist of cluster names forwarded by the webhook bridge.

Event flow

Event-driven issue detection: HyperPod emits cluster-state, node-health, and capacity events to Amazon EventBridge. The webhook bridge Lambda drops routine Info-level noise, maps the rest into a DevOps Agent investigation payload, signs it with HMAC-SHA256 using a shared secret stored in AWS Secrets Manager, and POSTs it to the agent’s generic webhook.

Polling-based issue detection: A periodic-audit Lambda checks Kubernetes state (CrashLoopBackOff pods, NotReady nodes) every 15 minutes and fires only when it finds a real issue, plus a daily heartbeat confirming the pipeline is alive. On a healthy cluster, nothing is POSTed, so no investigation runs and no cost is incurred.

Investigation: DevOps Agent receives the payload and runs two custom skills: the triage skill decides whether to link (duplicate), skip (noise), or proceed (investigate). The RCA skill reconstructs the timeline using describe-cluster, list-cluster-nodes, list-cluster-events, kubectl, and CloudWatch logs (HMA health monitoring, lifecycle scripts), then classifies the incident as Suppress, Monitor, Escalate, or Resolved.

Notification: An Amazon Lambda function sends notification emails via Amazon SES. It listens on the aws.aidevops event stream for investigation completions, reads the verdict from the agent’s journal, and sends an email with the headline, what happened, likely cause, and recommended action. Suppress verdicts are filtered to avoid noise on healthy clusters.

Getting started

For a step-by-step walkthrough to deploy this solution, visit the DevOps Agent Integration guide. Once you have the solution running, the following sections explain how to customize detection, reasoning, and notifications for your environment.

Prerequisites

An AWS account with AWS CLI v2 configured for the target region.

An existing SageMaker HyperPod cluster (EKS or Slurm orchestrator).

IAM permissions to create roles, deploy CloudFormation, manage Secrets Manager, and call devops-agent:* and eks:CreateAccessEntry.

For email notifications: a verified Amazon SES sender identity. You can verify an email address in the Amazon SES console or with the AWS CLI. After running the command below, the address owner will receive a verification email and must click the confirmation link:
```
aws ses verify-email-identity --email-address [email protected]
```
Recipients must also be verified if your SES account is still in sandbox mode.

Deploying with CloudFormation

The solution deploys as a single CloudFormation stack. Clone the awsome-distributed-ai repository, create a params.json with your cluster name and email settings, and run:

cd 1.architectures/5.sagemaker-hyperpod/tools/devops-agent

# 1. Set up a Python env with boto3 >= 1.43.25
python3 -m venv .venv && source .venv/bin/activate && pip install 'boto3>=1.43.25'

# 2. Fill in your cluster name and email addresses
cp deploy/params.example.json deploy/params.json
# edit: HyperPodClusterName, EmailSender, EmailRecipients 

# 3. Deploy
make deploy

This provisions the Agent Space with read-only EKS access (auto-discovered from the cluster’s orchestrator ARN), the EventBridge rule and webhook bridge Lambda, the periodic-audit scheduler, and the email notifier. For Slurm-orchestrated clusters, the EKS access step is skipped automatically.

The webhook bridge — mapping HyperPod events to DevOps Agent

An EventBridge rule captures HyperPod events and invokes a Lambda function. The Lambda forwards all Warn and Error level events, normalizing each into a DevOps Agent investigation payload. It extracts the failure message, instance group, and event metadata, then signs it with HMAC using a shared secret stored in AWS Secrets Manager, and POSTs it to the agent’s generic webhook endpoint. Info-level events are dropped at the bridge to avoid creating investigations for routine status updates.

A cluster allowlist parameter lets you scope which HyperPod clusters trigger investigations, useful when multiple clusters share the same account and region.

How the skills are defined — teaching the agent HyperPod’s operational model

AWS DevOps Agent skills are plain-English instructions that teach the agent how to reason about a domain. This solution includes two complementary skills:

Triage skill — LINKED / SKIPPED / PROCEED (view the skill document)

The triage skill runs first on every incoming task. It decides whether to link the event to an existing investigation, skip it, or proceed to a full investigation.

Why triage matters — a concrete example: When a single node fails, HyperPod’s replacement process emits multiple events in quick succession: “lost orchestration-ready status,” “provisioning started,” “capacity request initiated.” Without triage, each event would spawn a separate investigation. The triage skill recognizes these events belong to the same incident (same instance group + overlapping time window) and links them, so only one investigation runs. This saves investigation compute and avoids duplicate emails.
When to SKIP: When a node is already being replaced and a follow-up “lost orchestration-ready status” event arrives with a generic “Request to service failed” message, the triage skill recognizes that a replacement is already in progress for that instance group and skips the event. No new investigation is created for what is simply a progress update of an existing recovery.

RCA skill — timeline reconstruction and verdict (view the skill document)

When triage produces PROCEED, the RCA skill takes over. It reads cluster state, events, and logs, reconstructs an incident timeline, and classifies the situation into one of four verdicts:

RCA Flowchart showing the four phases of root-cause analysis: data gathering, timeline reconstruction, classification, and recurrence check

Phase 1 — Data gathering: The skill reads describe-cluster, list-cluster-nodes, list-cluster-events, and CloudWatch log streams (HMA health monitoring, lifecycle scripts) to collect the raw facts.
Phase 2 — Timeline reconstruction: It orders events chronologically and identifies the fault chain: what triggered what, which nodes were affected, and what recovery actions HyperPod took.
Phase 3 — Classification: Based on the timeline, recurrence statistics, and HyperPod’s resiliency behavior, it assigns a verdict:
- Suppress — a non-issue (for example, a transient event that has already resolved).

- Monitor — recovery is in flight; here’s the expected resolution window.

- Escalate — you need to act; here’s the root cause and recommended action.

Resolved — auto-recovery closed the loop; no action needed.
Phase 4 — Recurrence check: The skill computes sliding-window statistics over the one week cluster event history. When thresholds are crossed, the verdict escalates to alert the operator of a systemic pattern. For example, the same GPU error signature on the same instance group three or more times in a week, or five or more replacements fleet-wide in 24 hours.

The verdict is written to the agent’s investigation journal along with a human-readable report containing what happened, the likely cause, and recommended operator actions.

The periodic-audit Lambda — Kubernetes state monitoring

The periodic-audit Lambda fires every 15 minutes and inspects Kubernetes Pod/Node state directly (via the EKS API server). It checks for:

Pods in CrashLoopBackOff (default: flagged when restart count reaches five and the last crash is within 15 minutes)

NotReady nodes (default: flagged when a node has been NotReady for at least 15 minutes and at least 10% of nodes are affected)

Namespace-aware filtering controls which pods are checked:

Pods in kube-public and kube-node-lease are ignored entirely by default.

Pods in kube-system, aws-hyperpod, and amazon-cloudwatch are tagged as system-workload issues (distinct from user-workload issues in the verdict).

All thresholds and namespace lists are configurable via the CloudFormation stack parameters.

The Lambda POSTs a webhook event to DevOps Agent only when a real issue is found. On a healthy cluster, nothing is POSTed, so no investigation runs and no cost is incurred. A separate daily heartbeat schedule confirms the monitoring pipeline itself is alive. The heartbeat is visible in the DevOps Agent console but deliberately not emailed on healthy runs — so silence in your inbox means the cluster is healthy, not that the pipeline is broken.

Note: HyperPod infrastructure faults (node health, capacity errors, lifecycle-script failures) are handled event-driven by the webhook bridge. They come from the native HyperPod event stream in EventBridge. The periodic audit deliberately does not duplicate that path; it only covers Kubernetes workload state, which is not in the HyperPod event stream.

Closing the loop — the email notifier

An EventBridge rule on the aws.aidevops event stream captures investigation lifecycle events. The email-notifier Lambda processes these events through the following steps:

Event filtering: Only “Investigation Completed” events are processed (one email per investigation lifecycle). The event payload contains the agent_space_id, task_id, and execution_id.
Dedup: The Lambda checks an S3 marker at s3://<bucket>/emailed/<execution_id>. If present, this investigation has already been emailed and the event is dropped. This prevents duplicate emails when the same completion event is re-emitted.
Fetching the investigation context: The Lambda calls two DevOps Agent APIs:
- get_backlog_task(agentSpaceId, taskId) — retrieves the task metadata (title, priority, timestamps).
- list_journal_records(agentSpaceId, executionId) — retrieves the investigation’s findings, symptoms, and investigation gaps from the agent’s journal.
Suppress-verdict filtering: If the investigation produced a Suppress verdict or no findings at all, no email is sent.
Email composition: The Lambda composes a single HTML email from the journal records: a short headline followed by a one-paragraph summary covering what happened, the likely cause, and the recommended action.
Send via SES: The formatted email is sent to the configured recipients. After successful delivery, the S3 dedup marker is written.

The operator also has access to the full investigation in the DevOps Agent web console (see following “Viewing investigations” section).

Viewing investigations in the DevOps Agent console

For readers new to AWS DevOps Agent, here’s how to navigate to your investigations:

Open the AWS DevOps Agent console.
Select your Agent Space (named hyperpod-<cluster-name>-devops-agent by default).
From the Launch web app drop-down, choose an option to open the DevOps Agent web app.
Select Incidents from the left navigation pane to open the Incident Response Dashboard. It lists all investigations with their subject, status, and timestamp.
Select any investigation to see its full timeline, journal records, and the verdict report.

Asking the agent directly — the DevOps Agent Chat UI

Beyond the automated emails, you don’t have to wait for the next investigation to get answers about your cluster. You can open the DevOps Agent’s AI chat at any time and ask follow-up questions in plain English. The agent answers from the live cluster state, the investigation history, and the skills it has been taught.

For example:

“I got an email about a GPU failure in my cluster. Did it get resolved now with HyperPod’s resiliency?” — The agent checks the current cluster state, confirms whether the replacement succeeded, and provides a timeline of what happened (HMA detection → replacement initiated → node back in service), along with anything to watch for.

“Are there unhealthy Pods on my cluster?” — The agent inspects the Kubernetes state and reports any CrashLoopBackOff pods or NotReady nodes.

“I just triggered scaling up. Check if it is progressing well.” — The agent looks at the cluster’s current node counts vs. target counts and reports whether provisioning is on track.

AWS DevOps Agent chat interface showing a natural language query about cluster health

The chat conversations are stored per Agent Space, so you can revisit past interactions alongside the automated investigations. This makes the Agent Space a single pane of glass for both automated incident response and ad-hoc troubleshooting of your HyperPod cluster.

Extending the solution — detection vs. reasoning

The solution has two extension points, which serve different purposes:

Extending detection (what conditions are caught):
- Event-driven path: The webhook bridge Lambda drops Info-level events and forwards all Warn and Error level HyperPod events to DevOps Agent. This typically does not need modification. It already catches all actionable events.
- Polling-based path: The periodic-audit Lambda checks Kubernetes state. To detect additional conditions (for example, GPU allocation below a threshold or specific Pod labels stuck in error states), add that logic to the Lambda code.
Extending reasoning (how the agent investigates and classifies): edit the plain-English skill definitions. For example, you can teach the RCA skill new classification rules, add domain-specific context about your workload’s expected behavior, or adjust the recurrence thresholds.

Detection is code; reasoning is natural language. Both are in the repo and designed to be customized independently.

Investigation feedback

After each investigation completes, a Feedback button appears in the DevOps Agent console. Clicking it opens the Investigation feedback dialog, where you can:

Rate whether the root cause was correct

Indicate whether human steering was needed during the investigation

Provide written feedback explaining what could be improved

This structured feedback is stored per investigation. An auto-learning mechanism that uses this feedback to improve future investigations is actively being developed.

DevOps Agent APIs used by this solution

For readers interested in the programmatic integration, here are the key DevOps Agent APIs this solution calls:

Component	API	Purpose
Webhook provisioner (deployment)	register_service	Register the generic webhook service with DevOps Agent
Webhook provisioner (deployment)	associate_service	Associate the webhook with the Agent Space
Skill uploader (deployment)	list_assets	Check if a skill already exists
Skill uploader (deployment)	create_asset / update_asset	Upload or update the triage and RCA skill definitions
Email notifier (runtime)	get_backlog_task	Retrieve task metadata (title, priority, timestamps)
Email notifier (runtime)	list_journal_records	Retrieve findings, symptoms, and gaps from the investigation journal
Teardown	disassociate_service / deregister_service / delete_asset	Clean up on stack deletion

Cleaning up

To remove all resources created by this solution, run:

make teardown-stack

This deletes the CloudFormation stack, removes the Agent Space, EKS access entries, secrets, and email configuration.

Additionally, if you no longer need the prerequisite resources, you can revert their setup, for example, deleting the verified Amazon SES email address identities you created for notifications.

Cost considerations

This solution is designed to be near-zero cost on a healthy cluster and scales proportionally with fault volume. Cost scales with fault volume, not node count directly. At large scale (100+ nodes), the triage skill becomes critical. A single hardware fault can generate 5-10 correlated EventBridge events, most of which are filtered by the webhook bridge Lambda before reaching the agent. Where triage adds value is linking and deduplicating across similar faults that affect multiple instances, or repeated faults on the same instance over time, consolidating them into a single investigation instead of many. As an example, a 500-node training cluster might see 20-50 investigations per month after filtering and deduplication.

Filtering and triage are your cost savers at scale. The webhook bridge filters correlated events from a single node failure (5-10 EventBridge events reduced to 1 forwarded event), eliminating redundant investigations at the source. Triage then links similar faults across multiple instances into a single investigation. For example, if 5 nodes hit the same GPU error in a window, triage consolidates them into 1 investigation instead of 5 (saving 4 × $4 = $16). The bigger the cluster, the more both layers save.
Investigation duration grows sub-linearly. A 1000-node cluster investigation doesn’t take 100x longer than a 10-node one. The agent queries describe-cluster and list-cluster-events once regardless of size. The data returned is bigger, but the API call count is similar.
CloudWatch Logs queries are the variable. On large clusters, the agent may query more HMA log streams, which takes longer agent-seconds AND incurs CloudWatch Logs Insights charges on your account (not part of DevOps Agent pricing).

DevOps Agent (the primary cost driver): Estimates based on 2 accelerator instances in a cluster

Component	Pricing	Your cluster estimate
Investigations	$0.0083/agent-second	~$4/investigation (at 8 min avg)
Chat (on-demand SRE tasks)	$0.0083/agent-second	~$0.25/chat query (at 30 sec avg)
Daily heartbeat	$0.0083/agent-second	~$1-2/day (short investigation confirming health)

On a healthy cluster with no faults, only the daily heartbeat fires, approximately $30-60/month in DevOps Agent time. On a cluster experiencing 5 real faults per week (typical for a large GPU fleet), expect ~20 investigations/month × $4 each = $80/month in investigation costs.

Free tier and credits:

New DevOps Agent customers receive a 2-month free trial (20 hours of investigations, 20 hours of chat per month). Enterprise Support customers receive monthly credits equal to 75% of their AWS Support charge toward DevOps Agent usage.

Supporting infrastructure (secondary costs):

Component	Monthly Cost Estimates
Lambda invocations	~96/day (15-min audit) + event-driven = well within free tier
S3 (skills + dedup markers)	< $0.01 (a few MB total)
Secrets Manager (1 secret)	$0.40
EventBridge rules	Negligible (per-event pricing)
SES emails	$0.10/1000 emails — at most 1 per investigation
CloudWatch Logs (Lambda)	< $1 (minimal log volume)

Total estimated monthly cost:

Scenario	DevOps Agent	Infrastructure	Total
Healthy cluster (no faults)	~$30-60 (heartbeat only)	< $2	~$32-62/month
Moderate faults (5/week)	~$80-120	< $2	~$82-122/month
Heavy faults (20/week)	~$320-400	< $5	~$325-405/month

How cluster size impacts cost

Factor	Small cluster (1-10 nodes)	Large cluster (100-1000 nodes)
Fault frequency	Rare (maybe 1-2/week)	Constant (NVIDIA reports ~1 fault/2-3 hours at 10K GPU scale)
Events per fault	Few (1 node replacement = 3-5 events)	More (cascading replacements, capacity queuing)
Investigation duration	Shorter (less state to read, fewer events in timeline)	Longer (more nodes to describe, more events to correlate, larger CloudWatch log groups to query)
Triage value	Low (few duplicates)	High (one fault generates many correlated events — triage links them into 1 investigation)
Periodic audit	Fast (few pods/nodes to check)	Slower (more K8s state to inspect)

Cluster Size	Faults/month	Investigations	Est. Agent Cost
1-10 nodes (your test)	2-5	2-5 + heartbeat	$8-20/mo + ~$30 heartbeat
10-50 nodes (typical prod)	5-20	5-15 (triage dedup)	$20-60/mo + ~$30 heartbeat
100-500 nodes (large training)	50-200	20-50 (heavy triage)	$80-200/mo + ~$45 heartbeat
1000+ nodes (frontier)	200-700	50-100 (massive dedup)	$200-500/mo + ~$60 heartbeat

Cost control levers:

Disable the periodic audit (EnablePeriodicAudit: false) to eliminate the heartbeat cost. Live event bridging still works.
Triage (LINK/SKIP decisions) runs at task creation time. No investigation cost is billed for deduplicated or skipped events.
Suppress verdicts filter email notifications but the investigation still runs. If you want to eliminate that cost, tune your EventBridge rule to drop more event types at the bridge level.

Comparison to manual monitoring:

Without automation, each fault requires an on-call engineer to manually correlate events across CloudWatch, EKS, and the SageMaker console, typically 30-45 minutes of triage before they even know whether HyperPod is self-healing or needs intervention. This solution delivers a root-caused verdict in minutes at ~$4 per investigation, while providing 24/7 coverage without human wake-ups. The cost savings compound with cluster scale: at 20 faults per month, that’s 10-15 hours of engineering triage replaced by automated verdicts.

Conclusion

In this post, we showed how to build an end-to-end agentic incident-response pipeline for SageMaker HyperPod using AWS DevOps Agent. The solution complements HyperPod’s built-in resiliency by watching for the operational conditions where a human still wants to be in the loop: configuration issues affecting provisioning, capacity-bound recoveries, recurring hardware fault patterns, and workload-level conditions. It delivers clear, root-caused verdicts to the operator’s inbox.

The broader takeaway is a reusable pattern: teaching an AI agent a domain’s operational model through plain-English skills, so it can distinguish “the system is recovering on its own” from “this needs a human decision.” This pattern applies beyond HyperPod to any event-driven AWS service where operational conditions benefit from automated correlation and triage.

What’s next

To deploy the solution, follow the step-by-step instructions in the DevOps Agent Integration guide on the AI on SageMaker HyperPod site. Once it’s running, you can customize it for your environment:

Adjust the CloudFormation parameters: tune the periodic-audit schedule, CrashLoopBackOff thresholds, NotReady node percentages, namespace filtering, and email recipients. No code changes required.

Extend detection: modify the periodic-audit Lambda to check for additional Kubernetes conditions specific to your workloads (for example, GPU allocation below a threshold, specific Pod labels stuck in error states).

Extend reasoning: edit the triage or RCA skill definitions to adjust classification rules, add domain context about your expected cluster behavior, or tune the recurrence thresholds.

Add notification channels: connect Slack or PagerDuty via DevOps Agent’s built-in integrations or via a sibling EventBridge rule on the same aws.aidevops event stream.

The skills are plain English. Iterate on them the same way you’d iterate on a runbook.

About the authors

Accelerating AWS Network Firewall troubleshooting with AWS DevOps Agent

2026-07-24 Salman Ahmed

Post Syndicated from Salman Ahmed original https://aws.amazon.com/blogs/security/accelerating-aws-network-firewall-troubleshooting-with-aws-devops-agent/

When an administrator introduces a rule change in AWS Network Firewall and network connectivity is disrupted, pinpointing the cause requires inspecting multiple points in the traffic path. The firewall gives you stateless and stateful rule engines, domain rules, and routing to the firewall endpoint inside your Amazon Virtual Private Cloud (Amazon VPC). A network drop looks the same from the workload no matter where it started. Isolating the cause means correlating the alert and flow logs with the firewall configuration, route tables, and recent API calls in AWS CloudTrail that might have changed them. That manual correlation is exactly where AWS DevOps Agent helps, accelerating root cause analysis so you can restore connectivity in minutes instead of hours.

AWS DevOps Agent does that correlation for you. As your always-available operations teammate, it resolves and proactively prevents operational issues across AWS, multicloud, and on-premises environments. When an Amazon CloudWatch alarm triggers, it reaches the agent through a webhook. The agent then reads the firewall configuration and logs through AWS APIs, ties the drop to recent API activity, and returns a root cause with a mitigation plan you review before you apply it.

This post connects CloudWatch monitoring to DevOps Agent. It walks through three Network Firewall failures from end to end. The first is a domain deny list blocking a legitimate endpoint. The second is a stateless rule priority misconfiguration. The third is an asymmetric cross Availability Zone (AZ) routing drop. Each maps to a different layer, so each leads down a different investigation path. An AWS Cloud Development Kit (AWS CDK) app deploys the whole environment in your own account so you can reproduce each failure and follow along.

The sample workload

As part of this blog post, we provide a CDK stack that deploys both the AWS DevOps Agent Space and a sample workload used to walk through three separate troubleshooting scenarios. A single t3.micro instance in a protected subnet checks its connectivity to a test endpoint on a continuous loop and publishes results to CloudWatch. Traffic takes the internet egress path through Network Firewall, the NAT gateway, and the internet gateway, so the firewall can intercept or drop it. After completing the walkthrough, you can apply the same troubleshooting techniques with DevOps Agent against your own Network Firewall deployments.

The test endpoint runs in a separate VPC deployed by the same CDK app. It serves HTTPS on port 443 and TCP on port 9142, giving each scenario a different protocol layer to exercise: Scenario 1 targets a TLS connection on 443 (matched by Server Name Indication), Scenario 2 targets a TCP connection on 9142, and Scenario 3 exercises the whole egress path.

A live status page shows one card per scenario plus the network topology. The whole stack deploys from a single CDK app across two Availability Zones, each with a firewall endpoint and NAT gateway, which is what makes Scenario 3 possible.

As shown in the following figure, the egress data path runs from the workload through Network Firewall and the NAT and internet gateways to the test endpoint. The alarm pipeline runs from CloudWatch through Amazon Simple Notification Service (Amazon SNS) and the webhook AWS Lambda function to DevOps Agent.

Figure 1: The sample workload

To use this with your own workload, you need a CloudWatch alarm that detects the connectivity problem and the webhook pipeline (SNS topic and Lambda function) that delivers it to DevOps Agent. The agent reads your firewall configuration, logs, and CloudTrail through AWS APIs, so no additional instrumentation is needed on the firewall side.

Prerequisites

To follow along with this post, you need:

An AWS account with permissions to deploy Amazon VPC, Network Firewall, Amazon Elastic Compute Cloud (Amazon EC2), CloudWatch, Amazon SNS, Lambda, Amazon CloudFront, and AWS Identity and Access Management (IAM) resources
Access to AWS DevOps Agent, with permission to configure its webhook
Node.js 18 or later and npm installed
AWS Command Line Interface (AWS CLI) 2.x configured with credentials
AWS CDK 2.x is required. You can use it through the project’s npx dependency, or install it globally:
- npm install -g aws-cdk

Deploy the sample workload

Clone the project and deploy it into us-east-1 with one command (set awsRegion to use another AWS Region).

git clone https://github.com/aws-samples/sample-accelerating-aws-network-firewall-troubleshooting-with-aws-devops-agent.git
cd sample-accelerating-aws-network-firewall-troubleshooting-with-aws-devops-agent
bash scripts/deploy.sh

The script checks prerequisites, installs dependencies, compiles and tests, and bootstraps the CDK if needed. It then deploys all the stacks from a clean baseline and prints the outputs, including the status-page URL and sign-in details.

Open the status-page link (an https://<random-id>.cloudfront.net address).
Sign in using the username and password provided from the CDK output and confirm all three cards show the green Healthy status.
Keep the page open while you run the scenarios.

Connect AWS DevOps Agent

To connect AWS DevOps Agent to the alarm pipeline

In the AWS DevOps Agent console, open the nf-devops-agent-space Agent Space created by the CDK deployment.
Configure the DevOps Agent webhook and download the CSV file with the webhook URL and signing secret.
On the status page, choose Configure webhook, paste the URL and signing secret, and save. The page writes them to the nf-devops-agent-webhook-credentials AWS Secrets Manager secret, so there is no AWS CLI or console step. Until you set it, the bridge Lambda function sees a placeholder and skips delivery.

Verify the path before you run a scenario. In the Lambda console, open nf-devops-agent-webhook and use the Test tab with this event.

{
  "Records": [
    {
      "Sns": {
        "Message": "{\"AlarmName\":\"TEST-webhook-verification\",\"AlarmDescription\":\"[TEST] Webhook integration test - not a real alarm.\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"[TEST] Manual webhook connectivity test. Safe to ignore.\",\"Region\":\"us-east-1\"}"
      }
    }
  ]
}

A 200 response confirms the path, and a test investigation appears in the DevOps Agent Operator Web App view.

How the alarm pipeline works

Every scenario reaches DevOps Agent the same way. A CloudWatch alarm moves to ALARM and notifies the SNS topic. Amazon SNS invokes a Lambda function. The function reads the webhook URL and signing secret from Secrets Manager, signs an alarm payload, and POSTs it to the DevOps Agent webhook (as shown in Figure 1). Amazon SNS also provides delivery retries, fan-out to other subscribers, and cross-account publishing.

Prebuilt Network Firewall metric (Scenario 1) – Alarm-1 watches the DroppedPackets metric, summed across the stateful streams, and triggers when drops rise above a baseline threshold. This requires no workload or custom metric and works on an already-deployed firewall. However, it only tells you that the firewall is dropping packets, not which rule is responsible.
Application health metric (Scenarios 2 and 3) – Alarm-2 and Alarm-3 watch a custom metric from a connectivity check. Use this for an alarm tied to user-facing impact or to tell one traffic path from another, which requires running a component that emits the metric.

Alarm	Source	Triggers when
Alarm-1	`Native AWS/NetworkFirewall DroppedPackets`	The firewall’s dropped-packet count rises above the baseline
Alarm-2	Custom application health metric	The port 9142 (TCP) connectivity check to the test endpoint is being dropped
Alarm-3	Custom application health metric	The cross Availability Zone connectivity check is being dropped

Run the scenarios

Work through each of the scenarios one at a time, following the same cycle. Interrupt network connectivity, watch the alarm trigger, let DevOps Agent investigate, apply the recommended fix, and confirm recovery before moving on.

The status-page cards follow the live CloudWatch alarm state. A card shows a green dot and the word Healthy when its alarm is clear, and a red dot and the word DROPPED when its alarm triggers. In the DROPPED state the card also adds a Condition: line describing what’s being dropped, which isn’t shown when the card is healthy. Network Firewall applies changes to new flows, so a change shows within a minute or two. Recovery comes from the mitigation DevOps Agent recommends, which you review and apply.

Scenario 1. Domain deny list blocking a legitimate endpoint

At baseline, the rg-domain Suricata domain rule group denies only an unused placeholder, so the test endpoint stays reachable. The rule group inspects the TLS Server Name Indication (SNI) on each outbound connection and drops any that matches a denied domain. The exact rule syntax and console steps follow.

To add the domain deny rule

Go to the Amazon VPC console.
In the navigation pane, under Network Firewall, choose Network Firewall rule groups.
Choose the rg-domain rule group to open its details page.
In the Rules section, choose Edit.
The rules box already contains two baseline placeholder rules (they match blocked.placeholder.invalid, so nothing real is denied). Leave those in place. Find the <app-endpoint-dns> value for Scenario 1 in the deployment script output (a Nework Load Balancer (NLB) DNS name such as NfTest-AppNl-a1b2C3dEf4G5-1234abcd5678efgh.elb.us-east-1.amazonaws.com). On a new line below the existing rules, add a drop rule that matches that DNS name on the TLS SNI, then choose Save.
drop tls $HOME_NET any -> $EXTERNAL_NET any (ssl_state:client_hello; tls.sni; content:"<app-endpoint-dns>"; startswith; nocase; endswith; msg:"S1 domain denylist"; flow:to_server, established; sid:2000002; rev:1;)
After saving, the rules box holds all three lines. The two placeholders remain, plus the new drop rule for the endpoint DNS name (note the distinct sid 2000002).

Figure 2: Scenario 1 – Firewall rule change blocking the connection

What happens. The workload’s HTTPS check to the test endpoint times out, the “AWS/NetworkFirewall DroppedPackets metric climbs above baseline, and Alarm-1 moves to ALARM. The Scenario 1 card reads DROPPED (with the condition Firewall dropping the monitored domain on its allow/deny rules), while the Scenario 2 and Scenario 3 cards stay Healthy (Figure 3). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, because the packets are now dropped at the firewall. To demonstrate the resulting failure, the HTTPS · SNI line from the internet gateway to the test endpoint is shown in red, which the legend defines as dropped (root cause).

Figure 3: Scenario 1 active – Traffic blocked at the firewall

Let DevOps Agent investigate. The agent runs several lines of investigation in parallel and correlates them:

Reads the DroppedPackets metric and correlates the spike with a simultaneous drop in passed packets, confirming the firewall is actively blocking traffic.
Reads the ALERT log and finds the workload’s TLS connections to the test endpoint blocked by the S1 domain denylist rule.
Compares the current state against a baseline window, where the same endpoint was reachable with no alerts, which shows the block is new.
Searches CloudTrail and surfaces the UpdateRuleGroup call that added the deny rule, identifying the user, role, and timestamp approximately one minute before the drops began.
Reports the root cause as that manual rule-group change. Recommends removing the deny entry or adding an allow exception and enabling FirewallPolicyChangeProtection to prevent unauthorized changes.
Presents this as a plan you review and apply, not an automatic change.

In the DevOps Agent Operator Web App view, the agent first restates the Alarm-1 trigger and confirms the firewall is dropping packets above the threshold (Figure 4).

Figure 4: Scenario 1 – The symptom

Next, the agent identifies the root cause: a manual update to the rg-domain rule group that added a domain deny rule (SID 2000002) shortly before the alarm fired, blocking TLS connections to the ELB endpoint (Figure 5).

Figure 5: Scenario 1 – The root cause

Finally, the agent presents a mitigation plan, recommending you remove the problematic deny rule (SID 2000002) to restore connectivity (Figure 6).

Figure 6: Scenario 1 – The mitigation plan

Note: In a real-world environment, this type of rule typically exists for a reason. Before removing it, verify whether it was intentional but scoped too broadly. If so, refine the rule to block only unauthorized endpoints rather than removing it entirely.

Confirm recovery. Apply the change the agent recommends. After the deny entry is gone, DroppedPackets falls back to baseline, Alarm-1 clears, and the card returns to green. Move on to Scenario 2.

Scenario 2. Stateless rule priority misconfiguration

At baseline, the rg-stateless-priority stateless rule group keeps the allow rule at priority 100 and the drop rule at 200 for the test class, TCP destination port 9142. The workload opens a TCP connection to the test endpoint on this port. Lower priority numbers evaluate first, so the allow rule wins. This scenario uses port 9142 instead of 443 to demonstrate a stateless rule, which matches on the packet’s 5-tuple (protocol, ports, addresses) rather than application content.

Introduce the change. Invert the two rule priorities so the drop rule evaluates before the allow rule. This is the kind of change a rushed rule edit can introduce.

To invert the stateless rule priorities

Go to the Amazon VPC console.
In the navigation pane, under Network Firewall, choose Network Firewall rule groups.
Choose the rg-stateless-priority rule group to open its details page.
In the Rules section, choose Edit.
Raise the (Action: Pass) rule’s priority number so it sits after the (Action: Drop) rule, then choose Save. For example, change the (Action: Pass) rule from 100 to 300 (any number higher than the drop rule’s 200 works). You only need to move one rule, and using 300 avoids a clash with the drop rule that already sits at 200. Network Firewall evaluates the lowest priority number first, so the (Action: Drop) rule at 200 now wins for this traffic class, ahead of the (Action: Pass) rule at 300.

Figure 7: Scenario 2 – Rule priority change blocking the traffic class

What happens. The drop rule now wins, the TCP connection to the test endpoint on port 9142 times out, the StatelessRuleFailures metric climbs above baseline, and Alarm-2 moves to ALARM. The Scenario 2 card reads DROPPED (with the condition Stateless rules dropping the monitored traffic class), while the Scenario 1 and Scenario 3 cards stay Healthy (Figure 8). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, because the packets are now dropped at the firewall. To demonstrate the resulting failure, the TLS :9142 line from the internet gateway to the test endpoint is shown in red, which the legend defines as dropped (root cause).

Figure 8: Scenario 2 active

Let DevOps Agent investigate. A stateless drop happens before traffic reaches the stateful inspection engine, so it produces no ALERT log entries. The agent turns to configuration and flow logs instead:

Reads the stateless rule group state and finds the drop rule at the lower priority number, ahead of the pass rule, so the drop evaluates first.
Reads the flow logs and sees passed packets drop to zero within a minute of the change.
Searches CloudTrail and surfaces the UpdateRuleGroup call that inverted the priorities, identifying the user, role, and timestamp about a minute before the alarm.
Reports the root cause as that priority inversion. Recommends removing the redundant drop rule and managing the rule group through infrastructure-as-code (IaC) to prevent manual misconfigurations.
Presents this as a plan you review and apply, not an automatic change.

In the DevOps Agent Operator Web App view, the agent first restates the Alarm-2 trigger and confirms that a workload connectivity health check is failing because the firewall’s stateless rules are dropping egress (Figure 9).

Figure 9: Scenario 2 – The symptom

Next, the agent identifies the root cause, using the rule-group state and CloudTrail to pinpoint the conflicting DROP/PASS rules, where the new DROP rule’s lower priority number makes it match first (Figure 10).

Figure 10: Scenario 2 – The root cause

Finally, the agent presents a mitigation plan, recommending you remove the conflicting DROP rule at priority 200 to restore traffic flow (Figure 11).

Figure 11: Scenario 2 – The mitigation plan

Confirm recovery. Apply the change the agent recommends. After the allow rule is ahead of the drop rule again, Alarm-2 clears and the card returns to green. Move on to Scenario 3.

Scenario 3. Asymmetric cross Availability Zone routing drop

At baseline, the protected subnet in each Availability Zone routes its egress through the firewall endpoint in that same Availability Zone , and the matching return route uses that same endpoint. One endpoint sees both directions of the flow, so the stateful engine completes the handshake. The workload runs in the protected subnet in us-east-1a (CIDR 10.0.4.0/24), so at baseline its egress and its return both use the us-east-1a firewall endpoint.

Introduce the change. Make the flow asymmetric by sending egress out one Availability Zone endpoint while the return comes back through the other. This takes two route edits, and both are required. With only the first edit the flow can still complete, so the alarm will not trigger until both are saved. It makes no firewall-policy change, mirroring a real multi-Availability-Zone routing mistake.

To create asymmetric cross Availability Zone routing

Go to the Amazon VPC console and choose Route tables in the navigation pane.
Flip the egress. Select the NfNetworkStack/SampleVpc/protectedSubnet1 route table (the us-east-1a protected subnet, where the workload runs). On the Routes tab, choose Edit routes. Its 0.0.0.0/0 route currently targets the us-east-1a firewall endpoint. For the target, choose Gateway Load Balancer Endpoint and select the us-east-1b firewall endpoint, then choose Save changes.
Move the return. Select the NfNetworkStack/SampleVpc/publicSubnet2 route table (the us-east-1b public subnet, where egress now exits). Choose Edit routes, then Add route. For the destination enter the workload CIDR 10.0.4.0/24. For the target, choose Gateway Load Balancer Endpoint and select the us-east-1a firewall endpoint. Choose Save changes.

After both edits, a flow’s egress leaves through the us-east-1b endpoint while its return is directed to the us-east-1a endpoint. Neither endpoint sees the whole flow.

Figure 12: Scenario 3 routing change breaking the flow’s symmetry

What happens. A new connection leaves through one endpoint. Its return arrives at the other endpoint, which never saw the connection open, so the handshake fails. Unlike Scenarios 1 and 2, this affects the whole subnet, so all egress stops and Alarm-2 and Alarm-3 both move to ALARM. The AWS/NetworkFirewall DroppedPackets alarm (Alarm-1) stays quiet because no endpoint is making a drop decision. The flow is lost to asymmetric routing rather than counted as a firewall drop. This is why monitoring application connectivity matters. A routing fault is invisible to the firewall’s own drop counter. On the status page, the Scenario 2 card reads DROPPED (with the condition “Stateless rules dropping the monitored traffic class”) and the Scenario 3 card reads DROPPED (with the condition Return traffic dropped by asymmetric cross-Availability-Zone routing), while the Scenario 1 card stays Healthy (Figure 13). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, while the egress path from the firewall through the NAT gateway and the TLS :9142 and HTTPS · routing lines to the test endpoint turn red, which the legend defines as dropped (root cause).

Figure 13: Scenario 3 – The status page during a path-wide outage

Let DevOps Agent investigate. Both Alarm-2 and Alarm-3 fire in the same datapoint. DevOps Agent recognizes them as linked and merges them into a single investigation:

Reads the flow logs and sees bidirectional TLS connections stop abruptly, with only one-way traffic remaining and no flows reaching the established state.
Reads the firewall metrics and sees received and passed packets shift from one Availability Zone to the other at the moment of the change.
Calls DescribeRouteTables and finds the egress route pointing at one Availability Zone firewall endpoint while the return route points at the other.
Searches CloudTrail and surfaces the ReplaceRoute and CreateRoute calls by the same user, about a minute before both alarms fired.
Reports the root cause as that asymmetric routing change. Recommends restoring symmetric same-Availability-Zone routing so egress and return traverse the same endpoint.
Presents this as a plan you review and apply, not an automatic change.

A mitigation plan is a recommendation you review, not an automatic change, and the right fix depends on the intended design. Restoring symmetric routing can mean sending the workload subnet’s egress back through its own-Availability-Zone firewall endpoint (this sample’s architecture) or, in a design that doesn’t inspect this path, back through a NAT gateway. The agent infers a plausible target from what it can observe, so review the specific route it proposes against your intended topology before you apply it. (Connecting your pipeline or infrastructure-as-code, covered in the next section, lets the agent recommend the target that matches your design.)

In the DevOps Agent Operator Web App view, the agent restates the Alarm-3 (AsymmetricFlowFailures) trigger and confirms the workload’s egress to a monitored endpoint is being blocked by the Network Firewall (Figure 14).

Figure 14: Scenario 3 – The symptom

Next, the agent identifies the root cause: manual route table changes that created cross-AZ asymmetric routing through the network firewall, breaking its symmetric routing requirement (Figure 15)

Figure 15: Scenario 3 – The root cause

Finally, the agent presents a mitigation plan, recommending you restore symmetric routing by pointing protectedSubnet1‘s default route back to the same Availability Zone firewall endpoint, so one endpoint sees both directions of the flow again (Figure 16).

Figure 16: Scenario 3 – The mitigation plan

Confirm recovery. Apply the change the agent recommends, after checking the route target matches your intended design. After the workload subnet’s egress and return use the same Availability Zone firewall endpoint again, the control probe recovers, the alarms clear, and every card returns to green.

Further considerations

In production a single change can trigger several alarms at the same time, as Scenario 3 shows. DevOps Agent links related investigations and works them as one, so you review a single root cause. You can validate the linked findings or unlink an alarm to investigate it independently. If you would rather collapse alarms before they reach the agent, you can add correlation logic in the bridge Lambda function, buffering and grouping by firewall. You can also add email, Amazon Simple Queue Service (Amazon SQS), or HTTP subscribers to the SNS topic, or add the webhook Lambda function to a topic you already run. DevOps Agent produces a mitigation plan but does not change your environment on its own.

You can also give the agent more to work with. DevOps Agent connects to source repositories and CI/CD pipelines, integrating with GitHub (including GitHub Enterprise Server and GitLab Self-Managed through a private connection). It can associate AWS resources with deployments of AWS CloudFormation, AWS CDK, Amazon Elastic Container Registry (Amazon ECR) images, and Terraform. With deployed configuration and recent deployment events in view, the agent correlates the disruption against the change that introduced it and recommends a fix matching your intended design. For this sample, that means recommending the workload subnet’s own Availability Zone firewall endpoint rather than a generic symmetric path.

DevOps Agent also supports proactive incident prevention. It analyzes patterns across past investigations and delivers recommendations to prevent similar issues from recurring, including governance recommendations that strengthen deployment processes and pipeline controls. For Network Firewall rule changes, this means the agent can recommend guardrails for your CI/CD pipeline based on the classes of misconfigurations it has already resolved. You can access these recommendations through the Improvements page in the DevOps Agent Operator Web App.

Clean up

Clean up the environment with one command.

bash scripts/destroy.sh

It reverts any active scenario, runs cdk destroy for all stacks, and sweeps for stragglers by the Project = nf-devops-agent tag. The main cost drivers are the two Network Firewall endpoints, the NAT gateways (one in the main VPC for each Availability Zone, one in the test-endpoint VPC), and the test endpoint’s load balancers. Each of these bills at an hourly rate for as long as it’s provisioned, whether or not traffic is flowing, so a stack left running continues to accrue charges around the clock even while idle. Running the scenarios and tearing the stack down the same day limits the cost to a few active hours rather than days of idle hourly charges.

Conclusion

In this post, we showed you how AWS DevOps Agent accelerates troubleshooting for three common network firewall connectivity issues. The first was a domain deny list. The second was a stateless priority inversion. The third was an asymmetric cross-AZ routing drop. For each one, DevOps Agent investigated the drop and returned a root cause with a mitigation plan you approve before applying. The first scenario triggered on a prebuilt Network Firewall metric, and the other two on application health metrics. That shows both ways to alarm on a firewall problem through one pipeline.

The pattern isn’t specific to Network Firewall. The same flow fits any service that emits CloudWatch metrics and logs, such as AWS WAF, security groups, and network ACLs. Clone the sample repository to explore the solution, then apply what you learn to your own firewall, application, and alarms. For more details, see the AWS Network Firewall Developer Guide and the AWS Network Firewall pricing page. Start with the Getting Started with AWS DevOps Agent guide to connect your first webhook.

Optimized Monitoring for Hybrid Environments with ICT Solutions

2026-04-15 Michael Kammer

Post Syndicated from Michael Kammer original https://blog.zabbix.com/optimized-monitoring-for-hybrid-environments-with-ict-solutions/32839/

ICT Solutions is a managed service provider (MSP) specializing in fully managed IT Support, cloud, cybersecurity and more. Based in Liverpool, they offer IT support across the UK.

They work together with Zabbix Premium Delivery Partner Opensource ICT Solutions to make sure that their customers get solid insights into their environments.

The challenge

While a lot of companies realize the potential of hybrid environments as opposed to full-cloud environments, on-premise equipment (including local network and server equipment) is still a big part of what they do. It’s relatively easy to monitor cloud equipment with Zabbix proxies in the cloud, but not every customer has what it takes to run a Zabbix proxy on site.

ICT Solutions offers fully managed hybrid environments that include monitoring, so their customers have never had to worry about Zabbix proxies. As such, ICT Solutions has been running Raspberry PI 3 devices for years. Environments grow, however, and managing tens or hundreds of Zabbix proxies is something that can take time when not properly set up.

As an MSP, ICT Solutions looks after approximately 160 clients, 3,000 workstations and 1,300 network devices. These include firewalls, switches, access points, on-premise and hosted servers, network attached storage, CCTV, and door access – just to name a few. They have clients that they fully support, and clients that use them as an extension of their own IT teams.

The company also has a wide variety of templates and scripts set up in Zabbix, along with many dashboards so that when issues arise, they can see straight away where an issue exists or provide a more targeted fault-finding process. They also provide their clients’ IT departments with access to their Zabbix environment so they can visually display this on screens for purposes of working together.

The solution

With Zabbix environments growing over the years, Ansible was deployed and Semaphore was harnessed to keep things simple and manageable. This makes proxy management a breeze, as all the ICT team needs to do to deploy a proxy is have a field engineer install it and then push a button to install all the required software, which leads to the proxy being fully secured and automated into Zabbix.

Unfortunately, proxy performance was dropping over time. As monitoring needs got more extensive, the Raspberry PI 3 was showing its age, which led to Raspberry PI 5 devices being ordered and installed.

Another problem often attributed to Raspberry PI devices is their reliance on SD cards. SD cards are prone to failure when overloaded, which can become a problem as Zabbix stores its proxy database on the SD card.

Fortunately, Zabbix 7.0 introduced the “ProxyBufferMode=hybrid”, which allowed the ICT Solutions team to use the RAM of the Raspberry PIs instead of SD cards for the database. They now write the history metrics to the database on the SD card only in the case of a longer outage.

The results

The end result is a manageable and highly scalable setup that provides ICT Solutions and their customers with valuable insights into their hybrid environments as well as improved flexibility and enhanced security.

The post Optimized Monitoring for Hybrid Environments with ICT Solutions appeared first on Zabbix Blog.

Detect Issues in Your Zabbix Instance Before It’s Too Late

2026-03-25 Janis Eidaks

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/detect-issues-in-your-zabbix-instance-before-its-too-late/32741/

In this blog post, I will show you how to detect performance issues in your Zabbix instance – in advance!

You might be using Zabbix to monitor your infrastructure, devices, and applications, but are you also monitoring your own instance? It might seem unnecessary – after all, what’s there to monitor, right? Your instance just works, so everything is good. What else is needed?

Remember though, if your Zabbix database system runs out of disk space, data collection will come to a halt. If the data collectors are insufficient, the collected data will be inconsistent, and this will also affect the problem detection.

If you run out of cache space on your Zabbix server, depending on which cache is affected, your Zabbix server might crash immediately or experience degraded performance. A lot of things can go wrong, and you need to stay ahead of them! Here’s how.

Tune your database

If you are using the default settings for your database, you are missing out on significant performance improvements that are just unused! Your actual instance performance is tied to the database performance. If the database performance is low, you will have a degraded Zabbix monitoring experience as well.

Do at least minimal fine-tuning, only change the settings you understand: read the documentation, check the official Zabbix blogposts, Zabbix community forum, and perform testing. Of course, you can use every tool at your disposal to make it work, such as AI, but always test the settings in the test environment.

The database tuning is a complex task. Initial parameters that you could tune for the MySQL DB are these:

innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
optimizer_switch=index_condition_pushdown=off
innodb_buffer_pool_size= ~75% of RAM if only DB engine running or less if shared with other applications

For a PostgreSQL database, you can use online tuner PGTUNE for initial configuration:

https://pgtune.leopard.in.ua/

Monitor the Zabbix database

It is important to monitor your database. Zabbix offers several out-of-the-box options to monitor the most popular databases through different methods: either by Zabbix agent or Zabbix agent2, by ODBC checks, using Zabbix Java gateway or by HTTP checks. If an issue is detected, you will get a corresponding problem event. Don’t forget to manually update the old Zabbix templates to the current version after the Zabbix server upgrades.

Fig 1. Some of the available out-of-the-box templates for database monitoring

Of course, depending on the approach you have selected to monitor the database, you will need to do some additional steps for that to work. More information on how to configure it is available on the Zabbix integration page.

Fig 2. Example of the configuration required to monitor the MySQL database with Zabbix agent2

Monitor the Zabbix server

The next thing you should check is the Zabbix server host dashboards. In new instances, the Zabbix server host has already been included out of the box with two templates: Zabbix server health and Linux by Zabbix agent. If such a host has not been retained for some reason, now it’s time to create it and start monitoring your Zabbix server.

The Zabbix server health template uses Zabbix internal items that do not require any interface. The Linux by Zabbix agent template does require a running Zabbix agent on the Zabbix server system in order to gather the OS related metrics.

Fig 3. Zabbix server host with linked templates

Check the current state of your Zabbix server

Once you have such a host, go to the menu Monitoring > Hosts and use the main filter to find the Zabbix server host and select its Host Dashboards.

Select the Zabbix server health dashboard. Below, you will see the following pages under it – Performance, Processes, and Statuses.

Fig 5. Zabbix server health dashboard page: Performance

Check the cache utilization

In the Performance page, you can see the usage of Zabbix server caches. You should make sure that all caches except the history cache are at least ~ 50 % free. Technically, you can make the caches as large as possible; at worst, they will just be under-utilised. So, adjust the cache sizes accordingly.

Consequences of running out of configuration cache

If you add a lot of hosts in an automated way and have a relatively small or default configuration cache size [configcache], you could fill this cache quickly. The consequences of it are:

The Zabbix server will crash
The Zabbix server will be unable to start
The Zabbix server will not collect any data

You will also see a warning message in the Zabbix frontend:

Fig 6. Zabbix server health dashboard page when running out of config cache

If the config cache does not fill up instantly, the problem event will be generated shortly after, and matching action operations will be executed while the Zabbix server is still running, for example, notifying admins about the issue. In the screenshot below, you can see that one action operation step was executed before the server crashed.

When a Zabbix component is not working as expected, your best source of information is the log file, as it informs you about the issues. Here is the error message in the Zabbix Server log file below.

Fig 8. Zabbix server log file error: out of memory for config cache

The solution is very simple: just increase the configuration cache size (two times or more) and restart the Zabbix server. If you expect a significant increase in hosts in the near future, you can be more generous and allocate more memory. My current Zabbix server is monitoring approximately 400 hosts.

Fig 9. The system information of my Zabbix server

Consequences of running out of value and history cache

So, what happens if you run out of value cache? Zabbix server performance will degrade, and the frontend will become noticeably less responsive. Why is that? Value cache stores item values used for calculated items and evaluating triggers. Now, for each trigger calculation that does not contain an item metric in the history cache will be retrieved directly from the database.

Fig 10. Zabbix server health dashboard performance page for cache usage

The history cache stores historical data that will be written to the database. If it’s mostly full, it means you might have issues with your database performance – the Zabbix server is unable to write data fast enough to the database. This can trigger a cascading performance degradation with a negative feedback loop. In my case:

A full value cache leads to additional DB read queries
DB performance drops, which leads to slow historical data writes to the database
The history cache also starts to fill up
The data collection is delayed due to the full history cache
As more data is collected, database read queries retrieve more data, progressively worsening the cycle

Technically, it does not require your value cache to be 100% full to have this issue – if a lot of triggers use a long-time interval for evaluation, you could have a situation where your value cache is only 85% or 90% full, but the Zabbix server is unable to fit the required item history records in available memory.

The issue with running out of value cache will also be logged in the Zabbix server’s log file.

Fig 11. The Zabbix server log file with value cache error

The solution to this issue is simple: increase the value cache size and restart the Zabbix server.
If you monitor your Zabbix server with the health template, problem events will be automatically generated when:

Value cache works in a low memory mode
History cache utilization exceeds 75 %

Fig 12. The generated problem events about the value cache issue

The issue with the Value cache working in low memory mode can also be seen in the graph below. Here you can see how many historical item values were present in cache, and how many had to be retrieved from the database directly.

Due to the terrible performance of the untuned database, when my history write cache fills up the data collectors are throttled, causing a pileup of delayed item collection.

Slow database queries will appear in the Zabbix server log file.

Fig 15. The Zabbix server log file with slow query errors

The result of cache tuning and database tuning

Increasing the value cache only partly solved one issue. After database tuning, database performance has improved significantly:

The history cache is now empty
No more value cache misses
No more delayed items

Fig 16. The Cache usage, server performance, and value cache effectiveness graphs

After the Database tuning, the agent poller process and history syncer utilization also decreased to a low level.

Fig 17. Data collector and internal process utilization graphs

Tune the Zabbix server configuration parameters

Check the Processes page in the Zabbix Health dashboard and adjust the parameters accordingly. Only adjust the parameters that you understand. Changing the parameters arbitrarily can lead to the following:

Wasted resources without effective performance improvement
Reduced Zabbix server performance
Zabbix server crashes

For the data collectors, generally you require only a relatively small number of asynchronous data collectors, as they are very efficient, relatively larger number of synchronous data collectors. The graphs showing the utilisation of the gathering processes are extremely useful for determining which ones need to be increased – if they are close to 100% utilized, it’s now time for you to take action and add more.

Pitfalls of misconfiguration

Now, regarding the pitfalls of misconfiguration or lack of tuning. Here is a scenario: installed the Zabbix components, MySQL database without any configuration tuning, except the configuration cache to avoid the Zabbix server crashing immediately. The Zabbix server is monitoring around ~400 hosts. The Zabbix agent poller process and history syncers are utilized close to 100%, like in the Fig.17 before the tuning.

You might think that increasing both of these processes would improve the situation, for example, by doubling the count of them: more parallel agent processes should collect more data, and the more history syncers should write more data to the database.

After restarting the Zabbix server and checking the graph, both processes are close to 100% busy and the metric collection is significantly delayed. This is much worse.

Fig 18. Async data collector and internal process utilization graphs

By quadrupling both processes, the result is even worse, with significantly delayed item value collection.

Fig 19. Async data collector and internal process utilization graphs

So, what is happening behind the frontend? Just increasing the number of agent poller collectors and history syncers results in even worse performance. Seems counterintuitive, right: more data collectors should mean more data will be collected, and more history syncers – should allow more data to be written in parallel to the database.

However, increasing the data collector count in this specific situation will just make things much worse: you can collect more data at the same time, but will still face the same bottleneck: the database. Increasing the history syncers in this case makes the situation much worse, as more simultaneous queries to the database force it to slow down even further. So once again, tune your database engine and get more performance out of it.

Summary

You should monitor all of your Zabbix components and react when issues occur. Also make sure that you receive the notifications in your preferred media type, so you can act immediately. The complete list of what to monitor is more extensive, but this blog post should provide you with some examples and inspiration. It is always a good idea to react proactively rather than deal with the issues after they occur.

The post Detect Issues in Your Zabbix Instance Before It’s Too Late appeared first on Zabbix Blog.

Port Monitoring with the Zabbix Widget Switch

2026-03-18 Patrik Uytterhoeven

Post Syndicated from Patrik Uytterhoeven original https://blog.zabbix.com/port-monitoring-with-the-zabbix-widget-switch/32603/

If you’ve ever monitored network switches in Zabbix, you know perfectly well that the data is there. Interfaces are polled via SNMP, triggers fire when a port goes down, and events are logged. Technically, everything works.

But when you open a 24- or 48-port switch in Zabbix, you’re usually looking at a list of interface items or triggers. It’s accurate, but not visual. You still need to read through interface names to understand what’s happening. During an incident, that costs time.

Of course, you can create network maps for each switch, draw every port on it etc, but that takes a lot of time and is tedious if you have many devices.

That’s why we created the Zabbix Widget Switch.

Instead of presenting interface states as text, the widget renders a visual representation of the switch directly inside a Zabbix dashboard. Each port is displayed as a graphical element, color-coded according to its operational state.

Green means up.

Red means down.

Grey means disabled or unused.

You immediately see the health of the entire switch — without scrolling, filtering, or interpreting tables.

Designed for real environments

This widget is not just a static visual block. It was designed with real operational use in mind.

You can:

Reuse shareable switch profiles across devices
Save profile presets directly from the edit form
Configure rows, ports per row, SFP count, and port index start
Support mixed RJ45 + SFP layouts with realistic placement
Add port labels (uplinks, APs, user links, etc.)
Show a utilization heatmap overlay with configurable thresholds and colors
Display a live panel with IN/OUT sparklines, current utilization, 24h online state bar, and 24h errors/discards bars and trend summaries
Configure traffic/error/discard/speed item patterns
Use item-key suggestions in the edit UI for faster setup
Choose traffic unit display (B/s or bps)
Show switch summary context (CPU, uptime, serial, software, VLANs, monitoring state, maintenance badge)

This makes it practical for real-world deployments, not just lab environments.

If you manage dozens of similar switches, profiles save time and keep dashboards consistent.

If you run a NOC screen, the legend ensures clarity for everyone.

If you monitor mixed copper and fiber ports, SFP support makes the layout realistic.

It adapts to how your network is built.

Why it matters

Monitoring is about reducing reaction time.

When a user says, “My connection dropped,” you don’t want to search through interface lists. You want to open the dashboard and immediately see that port 17 is red.

In NOC environments, a visual layout is even more powerful. A quick glance at the screen tells you whether everything is healthy or if some uplink needs attention.

And during maintenance windows, a before and after check becomes an instant visual validation.

How it looks

A visual overview like this makes the state of a switch immediately obvious, even from across the room.

Open source

The Zabbix Widget Switch is open source and available for Zabbix 7.0 on GitHub:

https://github.com/OpensourceICTSolutions/zabbix-widget-switch

Feel free to test it, adapt it, or contribute.

I hope this widget will be helpful! If you have any questions or need help configuring anything on your Zabbix setup feel free to contact us at Opensource ICT Solutions.

Patrik Uytterhoeven

https://oicts.com

Disclaimer: Parts of this software were generated using Codex. We do not guarantee the total accuracy, security, or stability of the generated code.

The post Port Monitoring with the Zabbix Widget Switch appeared first on Zabbix Blog.

Showcasing Our Potential at Europol Industry and Research Days

2026-03-11 Michael Kammer

Post Syndicated from Michael Kammer original https://blog.zabbix.com/showcasing-our-potential-at-europol-industry-and-research-days/32733/

On February 24-26, Europol, the official law enforcement agency of the European Union. welcomed leading innovators, researchers, and law enforcement representatives to its headquarters in The Hague for the third edition of Europol Industry and Research Days.

This year marked the first time Zabbix met the criteria for event participation, an achievement that allowed us to showcase the benefits of Zabbix for law enforcement. Let’s take a look at the event, explore why Zabbix’s participation was a true milestone, and dive into the solutions Zabbix can provide for this rapidly growing vertical.

Onstage at Europol Industry and Research Days

The three-day event brought together Europol staff, representatives from law enforcement agencies in EU member states and Schengen-associated countries, private sector innovators, and research organizations. In total, 40 companies and eight EU-funded research projects were selected to present leading-edge technical solutions designed to address the evolving needs of European law enforcement.

Participants explored practical tools and emerging technologies via keynote speeches, short pitches, and in-depth live demonstrations. The event also served as a collaborative platform to strengthen the bond between law enforcement and the private sector, making sure that innovation keeps pace with increasingly complex security challenges.

Tops among 120 applicants

Zabbix’s participation in the event marks a significant achievement, as we were chosen from a group of more than 120 applicants to showcase our technology. It’s a strong public endorsement of our expertise and relevance in supporting mission-critical environments.

Our team demonstrated how robust IT infrastructure monitoring with Zabbix can enhance operational resilience, situational awareness, and system reliability — all essential components for modern law enforcement agencies.

A first for Zabbix – and Latvia

Zabbix’s presence at the Industry and Research Days also represents a milestone for Latvia. We are the first Latvian organization ever selected to participate in the event, highlighting both our technical leadership and Latvia’s growing role in the European cybersecurity and IT innovation landscape.

By contributing to discussions and live demonstrations, we reinforced our commitment to supporting secure and resilient digital infrastructures across Europe and highlighted the increasing importance of cross-sector collaboration in safeguarding Europe’s digital and operational environments.

Zabbix for law enforcement

By providing real-time monitoring and visualization of critical IT infrastructure, Zabbix allows law enforcement agencies to maintain full visibility over servers, networks, databases, and applications. At the same time, customizable dashboards and alerts allow operators to quickly identify performance issues, service outages, or abnormal behavior across complex environments.

When it comes to surveillance systems, Zabbix can monitor cameras, video management systems, storage devices, and network connectivity, making sure that surveillance infrastructure remains continuously operational and immediately alerting key personnel when cameras go offline, storage capacity is low, or network latency affects video streams.

Zabbix is also well suited for air-gapped environments, which are common in sensitive law enforcement and security infrastructures. Because it can be deployed entirely on-premise without relying on external cloud services, it enables secure monitoring of isolated networks while still delivering comprehensive metrics, alerts, and reporting.

Thanks to proactive incident detection and mitigation, Zabbix analyzes system metrics and triggers alerts when thresholds are exceeded or anomalies are detected. Automated notifications and integrations with response tools allow IT teams to react quickly and resolve issues before they disrupt operations.

Zabbix also supports compliance efforts (including requirements aligned with frameworks such as NIS2) by providing audit trails, monitoring logs, availability reports, and security-related metrics. These capabilities help agencies demonstrate operational oversight, risk management, and system reliability.

What’s more, APIs and webhooks allow Zabbix to easily integrate with existing law enforcement IT ecosystems, including ticketing systems, incident response platforms, SIEM solutions, and custom internal tools. This makes it a flexible component within a broader operational workflow, helping agencies centralize monitoring, automate responses, and maintain the reliability of mission-critical services.

Conclusion

Our participation in Europol Industry and Research Days marked an important step in expanding our collaboration with the European law enforcement community. By demonstrating how reliable, secure, and flexible infrastructure monitoring can support mission-critical operations, we highlighted the growing role of Zabbix in strengthening digital resilience.

The connections established and ideas exchanged during the event open the door to promising new collaborations, and we look forward to building on this momentum in the near future.

The post Showcasing Our Potential at Europol Industry and Research Days appeared first on Zabbix Blog.

Modernizing Public Service Monitoring with Zabbix and Prodemge

2026-02-25 Michael Kammer

Post Syndicated from Michael Kammer original https://blog.zabbix.com/modernizing-public-service-monitoring-with-zabbix-and-prodemge/32612/

Prodemge is the public IT company responsible for supporting the digital systems and services that drive the Government of Minas Gerais in Brazil. Its operations cover essential areas such as healthcare, education, public safety, finance, and infrastructure, ensuring that public policies reach citizens quickly, securely, and efficiently.

The challenge

Monitoring such a wide variety of IT environments and systems was becoming increasingly complex for Prodemge. The lack of a single source of information and real-time visibility made it difficult for teams to respond quickly to demands for innovation and improvements in digital services. This was an untenable situation, as public service monitoring supports strategic processes such as:

Contract tracking and supplier billing
Direct capacity monitoring by clients
Availability monitoring of telecom operator links
Measurement of system downtime integrated with third-party applications

The complexity increased with the adoption of hybrid cloud architecture, integration with government blockchain, relationships with critical service providers, and the role of telecommunications operators that connect the entire state infrastructure.

Given this context, it became necessary to reposition monitoring as a central element of the company’s technology governance, aligning processes, service performance, and institutional strategy.

The solution

The decision to adopt Zabbix for public service monitoring was made in 2023, when the tool was already present in part of the company’s infrastructure. In December of the same year, Target Solutions, a Zabbix Certified Delivery Partner, won the public bid and was contracted to begin the project. The implementation was structured around five main pillars:

Assessment and architecture. Integrations with cloud systems, container environments, legacy networks, and external services all needed to be mapped in order to guarantee security and compliance with public sector regulations.

Installation and configuration. More than 7,000 assets began being monitored, with around 20,000 items collected in real time. A total of 29 dashboards were developed, organized by technical areas, service layers, and criticality.

Internal training. Teams underwent training throughout 2024, focused on daily use of Zabbix, environment administration, and indicator analysis.

Integrations. Zabbix was integrated with data visualization tools, databases via ODBC, authentication systems, LDAP, corporate email, CMDB, service desk manager, the government network portal, service ticketing systems, change management modules, inconsistency detection tools, and internal APIs. Alerts began being sent via email, Telegram, and SMS, ensuring fast and traceable responses.

IT service management. One of the main advancements was IT service monitoring, especially the national identity card (CIN) service. This included:

Monitoring the application URL
Monitoring hosting servers
Integration with Federal Revenue Service and TSE data
Blockchain monitoring
Supervision of the supplier responsible for data processing

This model was also applied to other critical state services, including public safety and education.

The results

By monitoring more than 7,300 assets and collecting 865,000 items in real time with Zabbix, Prodemge repositioned monitoring as a pillar of IT governance, reducing incidents by 20%, strengthening contractual oversight, and consolidating a management model based on data and operational efficiency.

Currently, Prodemge’s production environment is 100% covered by Zabbix and includes the following:

7,301 monitored hosts
865,000 collected items
159 customized templates

The developed dashboards now directly support both technical and administrative management, providing views such as SLA monitoring for the administrative city complex, government network monitoring with visualization of the consumption of 2,284 links across more than 60 agencies, as well as dashboards dedicated to IT services, control of 635 active SSL certificates, and the data lake environment operated with the Cloudera platform.

As a result, there was an approximate 20% reduction in the number of opened incidents, mainly due to the mitigation of false positives, in addition to significant time savings in incident handling and event visualization by analysts and technicians.

Another concrete example occurred in the digital identity card service, where Zabbix identified connectivity failures in external integrations. After architectural adjustments, availability increased from 34% to 99% within one month.

Conclusion

With greater system integration and consistent data usage, Zabbix’s suitability for public service monitoring has made it a central part of Prodemge’s technical and administrative routine, modernizing infrastructure and ensuring greater system availability for the population.

The post Modernizing Public Service Monitoring with Zabbix and Prodemge appeared first on Zabbix Blog.

Monitoring the Stars with Zabbix and VIRAC

2026-02-10 Michael Kammer

Post Syndicated from Michael Kammer original https://blog.zabbix.com/monitoring-the-stars-with-zabbix-and-virac/32578/

The Ventspils International Radio Astronomy Center (VIRAC / VSRC) is a radio astronomy installation belonging to the Latvian Academy of Sciences.

It observes a wide variety of near-Earth and deep-space objects in the radio-wave spectrum, using RT-32 and RT-16 telescopes, which are parabolic antennas with diameters of 32m and 16m as well as a LOFAR phased antenna array.

Among its most notable ongoing projects is the establishment of cooperation with the Swedish Space Corporation (SSC) and participation in the European VLBI Network (EVN), where VIRAC performs joint simultaneous observations with similar stations worldwide.

We spoke with Arturs Orbidans, Head of the Engineering and Technical Operation Group at VIRAC, and Software Engineer Kristaps Blumbergs to find out how Zabbix keeps millions of Euros worth of high-tech equipment up and running.

What are the main tasks, objectives, and problems addressed by monitoring tools at VIRAC?

The main objective of our monitoring is to obtain values from the equipment used in radio-astronomical observations, such as the antenna control system, receivers (including cryogenic ones), a stable frequency source (active hydrogen maser), digitizers, and data recorders.

If any of these values deviate from the defined norm, or if a device reports an error state, engineers are notified via email so the issue can be resolved. In addition, the availability of all servers and computers located in Irbene is monitored and their parameters are tracked.

Why Zabbix? Was there a migration from another tool?

Previously, there was no single monitoring tool that did everything in one place – there were only methods for retrieving the required values or tools intended to monitor a specific server. This meant that extending or expanding the tooling was too complex, if not impossible.

We needed a solution that could monitor values from the required equipment in one place and notify engineers about errors. We chose Zabbix because it was already used in the VSRC High-Performance Computing (HPC) department, and Zabbix itself had been recommended to that department by the ITML department of the Ventspils University of Applied Sciences.

We’d like to ask about some Zabbix infrastructure specifics at VIRAC. How are the following used?

Zabbix proxy. There are plans to introduce a Zabbix proxy to reduce the load on the Zabbix server, as it is currently the only system collecting all data.

High availability. Not implemented at the moment, but we definitely have services where it would be necessary, for example the maser–GPS PPS signal delay reader, which determines the delay between the two signals with microsecond precision. This is important for defining an accurate time reference for observations.

Reports (Scheduled reports). One weekly scheduled report is used, which graphically shows changes over time in important parameters of the active hydrogen maser.

Scripts. None have been created yet, because for now the provided templates and the use of system.run() for obtaining other values are sufficient.

Overview of items (what is collected and how). Most items come from standard Linux/Windows server templates. Custom items very often use system.run(), which executes custom scripts for data collection. In addition, .json files are read and then split into multiple items.

Problem detection (what type of triggers are used and how complex they are). The created triggers are quite basic, since the obtained data is already closely tied to the actual equipment. Therefore, for most triggers associated with the created items, we check to see whether the value is equal to a specific value or whether a numeric value falls within a defined range.

Visualization (widgets and maps). From the built-in widgets, the graph and problems widgets are used. Shortly before the release of Zabbix 7.0, custom Zabbix widgets were developed, one of which is used to navigate Zabbix dashboards. This widget consists of two buttons with links to other dashboards.

The main widget displays the radio telescopes, with additional buttons placed at specific locations that indicate whether there are any problems with equipment in that particular area. For example, the laboratory button is placed on the radio telescope schematic at the location where the laboratory is located. It is shown in green when everything is fine and in red when a problem has occurred with one of the servers in that room.

This widget functions as a custom map, and when one of the buttons is clicked, another widget displays the values associated with the selected location. At the moment, all settings for the created widgets use constant values, so they cannot yet be dynamically applied to other use cases.

The described widgets can be seen in the image shown below, where Telescope Information is the above-mentioned “map,” and the Information Display widget shows the related items/values when one of the available buttons is pressed. Meanwhile, in the top-right corner, all key values are displayed for cases where there is no desire to click on specific buttons.

Is there a specific scheme for user roles or permissions?

There is no special user scheme, because our team is very small. It consists only of an admin user and guest users, who can view the custom widgets and see whether there are any problems.

What are your impressions after working with Zabbix?

So far, we have not encountered any problems and are very satisfied with Zabbix. In fact, Zabbix has saved several important scientific observations!

The post Monitoring the Stars with Zabbix and VIRAC appeared first on Zabbix Blog.

Best Practices for Deploying AWS DevOps Agent in Production

2026-01-29 Greg Eppel

Post Syndicated from Greg Eppel original https://aws.amazon.com/blogs/devops/best-practices-for-deploying-aws-devops-agent-in-production/

Root cause analysis during incidents is one of the most time-consuming and stressful parts of operating cloud applications. Engineers must quickly correlate telemetry data across multiple services, review deployment history, and understand complex application dependencies—all while under pressure to restore service. AWS DevOps Agent changes this paradigm by bringing autonomous investigation capabilities to your operations team, reducing mean time to resolution (MTTR) from hours to minutes.

However, the effectiveness of AWS DevOps Agent depends heavily on how you configure your Agent Spaces which control resource access boundaries. An Agent Space that’s too narrow misses critical context during investigations. One that’s too broad introduces performance overhead and complexity. This post provides best practices for setting up Agent Spaces that balance investigation capability with operational efficiency, drawing from our experience onboarding early customers and using DevOps agent across our own teams.

By the end of this post, you’ll understand how to structure Agent Spaces for optimal investigation accuracy, determine the right scope of resource access, and use Infrastructure as Code (IaC) to streamline deployment. Let’s start by understanding the foundational concept that makes all of this possible: the Agent Space itself.

What is an Agent Space and Why Does It Matter?

An Agent Space is a logical container that defines what AWS DevOps Agent can access and investigate. Think of it as the agent’s operational boundary—it determines which cloud accounts the agent can query, which third-party integrations are available, and who can interact with investigations.

Agent Spaces are critical because AWS DevOps Agent needs sufficient context to perform accurate root cause analysis.

When an incident occurs, the agent:

Learns your resources and their relationships across accounts
Correlates telemetry data from logs, metrics, and traces
Reviews recent changes including deployments and configuration updates
Generates and tests hypotheses by querying additional data sources

This view shows the key resources, entities, and relationships DevOps Agent has selected as a foundation for performing it's task efficently.

Figure 1: Agent Space Topology

If the Agent Space doesn’t include access to a critical account or integration, the agent might miss the root cause entirely. Conversely, an overly broad Agent Space introduces performance challenges as the agent considers more resource permutations during investigations.

Understanding these trade-offs between scope and performance is essential. The question becomes: how do you determine the right boundaries for your specific organization and operational model?”

Part 1: Design your Agent Space architecture

We recommend thinking about Agent Space boundaries the same way you think about on-call responsibilities: grant access to accounts relevant to the application, but separate production from non-production environments.

This approach provides several benefits:

Familiar mental model – Operations teams already understand on-call boundaries
Appropriate investigation scope – Mirrors how human engineers would investigate incidents
Two-way door decision – You can expand or narrow Agent Space scope as needs evolve
Performance balance – Provides sufficient context without overwhelming the agent

Determine Your Agent Space Boundaries

Start by mapping your application architecture to Agent Space boundaries and consider the following questions:

What defines a logical application?
- Does your team own multiple independent applications? If so, create separate Agent Spaces.
- Is it a monolith spanning multiple accounts? Then one Agent Space with cross-account access makes sense.
How do you organize on-call rotations?
- Separate teams for production versus non-production suggests separate Agent Spaces.
- One team handling all environments might work with one Agent Space per application.
What are your investigation patterns?
- Do production incidents require querying dependent services in other accounts? Include those accounts.
- Are environments completely isolated? Keep Agent Spaces separate.

Example decision tree:

Application: E-commerce Platform
├── Production environment
│ ├── Account 111111111111 (Frontend)
│ ├── Account 222222222222 (API Gateway + Lambda)
│ └── Account 333333333333 (RDS + DynamoDB)
├── Staging environment
│ └── Account 444444444444 (All resources)
└── Development environment
└── Account 555555555555 (All resources)

Recommended Agent Spaces:
→ "EcommerceProd" (accounts 111111111111, 222222222222, 333333333333)
→ "EcommerceNonProd" (accounts 444444444444, 555555555555)

Create one Agent Space per oncall team. The Production Oncall team manages the "EcommerceProd" Agent Space covering production accounts. The Non-Prod Oncall team manages the "EcommerceNonProd" Agent Space covering development and staging accounts. This 1:1 mapping provides operations teams with a familiar mental model where Agent Space boundaries match their existing oncall responsibilities.

Figure 2: Agent Space boundaries mirror on-call team responsibilities

Common Agent Space Patterns and Decision Points

Beyond the basic single-application pattern, organizations encounter more complex scenarios that require careful consideration. Here are critical patterns to address that we’ve seen customers successfully adopt:

Pattern 1: Investigations Spanning Multiple Teams. Large organizations with multiple teams (example: 3 teams managing 100+ production accounts) encounter situations where an issue originates in Team A’s infrastructure but the root cause lies in Team B’s services. The question becomes: how do you enable collaboration across Agent Spaces?

Recommended approach: Create application-specific Agent Spaces that include read-only access to shared resource accounts e.g. dependencies. Establish clear on-call escalation procedures and add them as runbooks when investigations identify cross-team root causes for efficient communication (e.g. via chat in Slack). Configure the shared service team’s resources with tags identifying which applications use them (example: app-id: ecommerce-frontend). Following a consistent tagging strategy provides investigation context for shared resources while maintaining clear resource ownership.

Pattern 2: Shared Services and Network Operations Center (NOC) Teams. Some organizations have centralized teams that provide and support shared infrastructure services (databases, networking, monitoring, security) used by multiple applications across the organization. These NOC or central operations teams need visibility into their services without requiring access to every application’s Agent Space.

Recommended approach: Create a dedicated Agent Space for the shared service team and configure an Agent Space scoped to the shared service team’s infrastructure and operational responsibilities:

Include AWS accounts containing shared databases, network infrastructure, centralized logging, and monitoring systems
Add relevant CloudFormation stacks for shared platform services
Configure IAM roles that provide read-only access to the specific resources the team supports
Include runbooks and operational procedures specific to the shared services

This follows the same principle as application-specific Agent Spaces: one Agent Space per on-call team, even when that Agent Space’s scope spans multiple applications. While shared services teams manage specific infrastructure domains, SRE teams often face an even larger challenge: operational responsibility for hundreds or thousands of applications at enterprise scale.

Pattern 3: Central Operations Teams Managing Many Applications. Central operations teams responsible for operational tooling across hundreds or thousands of applications can efficiently manage Agent Spaces at scale using Infrastructure as Code.

Recommended approach: Use the AWS CDK or Terraform samples available as starting points. These samples enable teams to:

Define a standardized Agent Space template with your organization’s required IAM roles, integrations, resource boundaries and governance tags
Deploy Agent Spaces programmatically as part of application onboarding workflows
Enforce compliance through AWS Config rules or service control policies
Track all Agent Spaces through consolidated billing and tagging (application-id, team, cost-center, environment)

Central operations teams manage the templates and governance policies, while application teams operate within those guardrails. This approach scales to thousands of applications with consistent configuration and automated deployment. AWS DevOps agent allows limiting agent access in an AWS account and controlling access for users to the operator console for teams to manage Agent Space access at scale.

A small platform team (a few engineers) manages 1,000+ Agent Spaces by maintaining standardized IaC templates (AWS CDK and Terraform). When new applications are registered, a CI/CD pipeline automatically deploys an Agent Space for that application team. This distributed pattern (one Agent Space per app team) scales to many applications without manual intervention, while maintaining investigation accuracy by avoiding a centralized "monitoring account" that would bias toward its primary application.

Figure 3: Enterprise scale pattern using Infrastructure as Code

Now that you understand how to design Agent Space boundaries aligned with your team structure and scale requirements, let’s walk through the practical implementation steps to bring these architectural patterns to life.

Part 2: Implement your Agent Space architecture

This section walks you through the practical steps of creating your first Agent Space—from verifying prerequisites and configuring IAM roles across accounts to integrating observability tools, setting up access controls, and testing your configuration to ensure investigations have the context they need.

Step 1: Agent Space Prerequisites

Before setting up your first Agent Space, ensure you have:

AWS accounts – At least one AWS account where your application resources run
IAM permissions – Sufficient access to create IAM roles and policies across accounts. AWS DevOps Agent requires two distinct sets of IAM permissions:
- Agent Space role permissions – The IAM role that AWS DevOps Agent assumes to query your AWS resources, access CloudWatch Logs, and discover topology. This role requires the AIOpsAssistantPolicy managed policy plus additional permissions for AWS Support and expanded capabilities. See the CLI onboarding guide for the complete role configuration.
- Operator app role permissions – The IAM role that controls what human operators can do in the AWS DevOps Agent web application, such as starting investigations, viewing results, and creating AWS Support cases. This role is separate from the agent’s investigation permissions.
Service Control Policies (SCPs) – Verify that your organization’s SCPs allow AWS DevOps Agent API actions. Common issue: Teams complete Agent Space setup but investigations fail because SCPs block aidevops:* actions or bedrock:InvokeModel actions. Review your AWS Organization’s SCPs and add exceptions for DevOps Agent if needed. Note that DevOps Agent and Amazon Bedrock inference are not impacted by policies that restrict customer content to specific AWS regions—Bedrock may use US regions other than US East (N. Virginia) for stateless inference.
Observability tools – At minimum, Amazon CloudWatch (automatically available via IAM roles) and Amazon CloudTrail. For comprehensive investigations, integrate Application Performance Monitoring tools like Datadog, Dynatrace, New Relic, Grafana, or Splunk. See Connecting telemetry sources for supported integrations.
Understanding third-party integration configuration – Some third-party tools require a two-step configuration process:
- Account-level registration – Tools that use OAuth (like GitHub, Dynatrace) must first be registered at the AWS account level through the DevOps Agent console. This establishes OAuth credentials that are shared across all Agent Spaces in your account.
- Agent Space-level association – After registration, each Agent Space individually specifies which resources from that tool to use. For example, after registering GitHub once, Agent Space “EcommerceProd” can associate only production repositories while Agent Space “EcommerceNonProd” associates development repositories.Other tools like Datadog, New Relic, and Splunk can be directly associated with an Agent Space using API keys or tokens without separate account-level registration. CloudWatch requires no additional configuration beyond IAM roles.
Source control – GitHub or GitLab repository access for code context and deployment correlation (optional but highly recommended)
IaC tooling – AWS CDK (TypeScript/Python), Terraform, AWS CLI, or AWS Management Console for Agent Space deployment

With prerequisites verified, you’re ready to create your Agent Space and establish the IAM trust relationships that enable investigations.

Step 2: Create an Agent Space

AWS DevOps Agent requires IAM roles in each AWS account within the Agent Space boundary. The agent assumes these roles to query CloudWatch Logs, describe resources, and build application topology.

The AWS DevOps Agent is designed to retrieve operational data from multiple AWS Regions across all AWS accounts that you grant access to within the configured Agent Space, enabling comprehensive visibility into distributed infrastructure and applications regardless of their geographic deployment, while supporting multiple accounts through a configuration process that involves creating IAM roles with appropriate trust policies and permissions in secondary accounts

Option A: Use the AWS Console wizard
Navigate to the AWS DevOps Agent console and choose Create Agent Space and follow the guided setup to create IAM roles in each target account.

The Create an Agent Space setup wizard in the AWS Management Console showing Agent Space Details.

Figure 4: Creating an Agent Space in the Console

The setup wizard helps in configuring cross-account trust relationships.

Shows the Agent Space Management Console and in particular the capability to configure your Agent Space to access multiple accounts.

Figure 5: Multiple account configuration for your Agent Space

Option B: Use Infrastructure as Code (Recommended)
We provide sample CDK and Terraform templates that automate Agent Space creation and IAM role deployment across multiple accounts.

AWS CDK example (TypeScript):

//If you have many accounts, use a loop:

const accounts = [
  { id: '111111111111', name: 'Prod', role: prodRole, stage: 'prod' },
  { id: '222222222222', name: 'Dev', role: devRole, stage: 'dev' },
  { id: '333333333333', name: 'Test', role: testRole, stage: 'test' },
];

accounts.forEach(account => {
  const association = new devopsagent.CfnAssociation(this, `${account.name}Association`, {
    agentSpaceId: agentSpace.ref,
    serviceId: 'aws',
    configuration: {
      aws: {
        assumableRoleArn: account.role.roleArn,
        accountId: account.id,
        accountType: 'monitor'
      }
    }
  });

  association.addDependency(agentSpace);
  cdk.Tags.of(association).add('stage', account.stage);
});

For detailed instructions on setting up IAM roles and permissions across accounts, see the CLI Onboarding Guide.

Once your Agent Space exists and has access to AWS accounts, the next critical step is connecting the observability and development tools that provide investigation context beyond AWS native services.

Step 3: Configure Integrations

AWS DevOps Agent investigates incidents by correlating data from multiple sources. The more context available, the more accurate the root cause analysis.

Recommended integrations by priority:

Amazon CloudWatch – Provides logs, metrics, and traces from AWS services. The agent queries CloudWatch Logs Insights automatically during investigations. No additional configuration is needed if IAM roles are properly configured.
Application Performance Monitoring tools – Datadog, Dynatrace, New Relic, and Splunk provide distributed tracing, custom metrics, and application-level context. Configure via Agent Space integrations in the AWS Console.
Code repositories – GitHub or GitLab integration enables the agent to review recent deployments and code changes. Requires OAuth or personal access token.
CI/CD pipelines – GitHub Actions or GitLab workflows help the agent correlate incidents with deployment timing. Configured alongside code repository integration.
Communication Channels – Slack and ServiceNow integration enables DevOps Agent to post real-time investigation updates to team channels and automatically update incident tickets with findings, root cause analysis, and recommended mitigation steps throughout the investigation lifecycle.

Advanced Integrations

Beyond built-in integrations, AWS DevOps Agent supports webhook triggered investigations and custom MCP (Model Context Protocol) servers so you can bring-your-own observability tools.

Webhook configuration for investigation triggers
Webhooks allow external systems (Grafana, Prometheus, PagerDuty, custom monitoring tools) to automatically trigger DevOps Agent investigations when incidents occur. Each Agent Space receives a unique webhook URL that accepts JSON payloads describing the incident.

Common configuration pitfalls:

Webhook authentication: Webhooks use HMAC signatures for security. Store the webhook secret in AWS Secrets Manager and rotate it according to your security policies.
Payload format: Ensure your monitoring tool sends incident context including timestamps, affected resources, and symptom descriptions. Richer context enables more accurate investigations.

For detailed webhook setup, see Invoking DevOps Agent through Webhook.

Bring-your-own MCP servers
If you use observability tools beyond the built-in integrations (Grafana, Prometheus, custom telemetry systems), you can connect them via MCP servers. MCP servers expose your tool’s data through a standardized protocol that DevOps Agent queries during investigations.

Key requirements for MCP servers:

Publicly accessible HTTPS endpoint: MCP servers must be reachable from the public internet. VPC-hosted servers are not currently supported.
Read-only tools only: For security, only expose MCP tools that perform read operations. Write operations introduce prompt injection risks.
Tool allowlisting: Register MCP servers at the account level, then selectively enable specific tools per Agent Space. Don’t grant access to all tools—choose only those relevant to investigations.

Common MCP setup errors:

Authentication misconfiguration: MCP servers support OAuth 2.0 or API key authentication. Verify your OAuth client credentials are correct and that token exchange URLs are accessible from AWS infrastructure.
Tool name length: MCP tool names have a maximum length of 64 characters. Longer names will fail registration.
Endpoint URL format: Use the full HTTPS URL including path. Example: https://mcp.example.com/v1/mcp not just mcp.example.com.

For comprehensive MCP server setup including authentication configuration, see Connecting MCP Servers.

Testing your integrations
After configuring webhooks or MCP servers, trigger a test investigation to verify connectivity:

For webhooks: Send a test payload from your monitoring tool and verify the investigation starts in the DevOps Agent web app
For MCP servers: Start an investigation manually and check the agent journal to confirm it successfully called your MCP tools
Review any errors in AWS CloudTrail logs which capture all DevOps Agent API calls including integration attempts

With your data sources connected, you now need to ensure the right people have appropriate access to investigations while maintaining security boundaries.

Step 4: Configure Access Controls

Agent Spaces support fine-grained access controls to ensure only authorized team members can interact with investigations.

Access control considerations:

Who should view investigations? Typically on-call engineers, SREs, and DevOps engineers. Consider including security teams for security-related incidents.
Who should create AWS Support cases? Typically on-call leads and senior engineers. Restrict this permission to prevent excessive case creation.
Who should modify Agent Space configuration? Typically central operations or infrastructure teams. Separate this from day-to-day investigation access.

IAM-based access control:

AWS DevOps Agent uses IAM policies to control access to Agent Spaces. Attach policies to IAM users, groups, or roles:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "devopsagent:GetAgentSpace",
        "devopsagent:StartInvestigation",
        "devopsagent:GetInvestigation",
        "devopsagent:ListInvestigations"
      ],
      "Resource": "arn:aws:devopsagent:us-east-1:123456789012:agentspace/EcommerceProd"
    }
  ]
}

AWS DevOps Agent operates within your AWS environment with privileged access to operational data across multiple accounts. While general security foundations apply, Agent Space configuration introduces specific considerations. For comprehensive security guidance, see the AWS DevOps Agent Security documentation.

Access controls are in place—now it’s time to validate that your Agent Space configuration provides the investigation coverage you need.

Step 5: Test and Iterate

Agent Space configuration is a two-way door decision. Start with a focused scope and expand based on investigation results.

Testing your Agent Space:

Trigger a test investigation using the AWS DevOps Agent web app.

Start an investigation and provide symptoms such as “High latency on /api/checkout endpoint”.
Observe which resources the agent queries.
Review investigation completeness. Did the agent identify the root cause?
Were any accounts or services missing from the investigation?
Did the agent have sufficient telemetry data?

Adjust Agent Space boundaries based on results.

Add accounts if investigations lack context.
Add integrations if telemetry gaps exist.
Narrow scope if performance degrades.

Conclusion

AWS DevOps Agent transforms incident response from a manual, time-consuming process into an autonomous, data-driven investigation. However, the agent’s effectiveness depends on proper Agent Space configuration. By following the on-call based approach—granting access to accounts relevant to your application while separating production from non-production environments—you provide sufficient context for accurate root cause analysis without introducing unnecessary complexity.

Key takeaways:

Think on-call boundaries – Agent Space scope should mirror how your team investigates incidents
Use Infrastructure as Code – CDK and Terraform templates ensure consistent, repeatable deployments
Integrate observability tools – More data sources equals more accurate investigations
Iterate based on results – Expand or narrow Agent Space scope as investigation patterns emerge

Next steps:

Create your first Agent Space

We’re committed to making AWS DevOps Agent easier to adopt and more accurate in solving customer problems. Your
Agent Space setup is the foundation for achieving fast, reliable incident resolution. Have questions or feedback? Leave a comment below.

Authors

Zabbix in 2025: A Year of Growth, Community, and Innovation

2026-01-20 Michael Kammer

Post Syndicated from Michael Kammer original https://blog.zabbix.com/zabbix-in-2025-a-year-of-growth-community-and-innovation/32470/

2025 has been a dynamic and crucial year for Zabbix — marked not just by global events and major releases, but also by meaningful community engagement, an important milestone in our history, new ways of sharing expertise, and headcount growth around the world – all while making sure our product evolves to provide even more value for our valued customers and partners. Let’s dig in!

Celebrating 20 years in business

On April 12, Zabbix officially celebrated its 20th anniversary as a company. It was a time for our entire community to step back, take stock, and imagine what the future might bring as our offices threw some amazing birthday parties to celebrate two decades of growing (and monitoring) together!

“I’m very satisfied with the results we’ve achieved as a company and I’m extremely grateful for our community, customers, and partners – they make everything we do possible.” – Alexei Vladishev, Zabbix Founder and CEO

A new path toward Zabbix expertise

In October, we launched Zabbix Academy, an online learning platform that offers a comprehensive library of interactive courses that are designed to help users master every aspect of Zabbix, on their own time and at their own speed.

Zabbix Academy includes everything from an introduction to open-source monitoring to advanced Zabbix security administration. There’s no need for any prior certification or training, and Zabbix Academy offers certifications and skill assessments aligned with Zabbix standards as well as a flexible pricing model that includes both free and paid courses and webinars, plus the ability to choose either a single standalone course or a yearly subscription.

“Our community is our greatest asset, and we believe that continuous learning is essential for long-term success, and Zabbix Academy is proof of that commitment.” – Kristine Lamberte, Head of Training at Zabbix

Nous sommes Zabbix!

In addition to all the other positive news, 2025 will be remembered as the year we announced the acquisition of our long-term partner IZI-IT and the establishment of Zabbix France, a new regional office dedicated to providing localized support and closer collaboration with French users and enterprises.

Headed by IZI-IT Founder and CEO Steve Destivelle, Zabbix France will leverage France’s strong technology ecosystem, skilled workforce, and strategic location to build a new European hub that will enable faster response times, support compliance with regional business and regulatory expectations, and ultimately boost our brand visibility across Europe.

Home sweet home

2025 also happened to be the year in which Zabbix finally outgrew our long-time Riga location. In September, our search for new premises led us to make the move to a larger and more suitable office space. The new office is spacious, flexible, easily accessible by bicycle or public transport, and features a modern infrastructure that can help us continue growing and competing in the global marketplace.

Building a better product

The big news on the product development front was the release of Zabbix 7.4 in July. Zabbix 7.4 introduces a wide variety of new features that users have requested, while delivering significant UI/UX improvements that make monitoring with Zabbix even more accessible and efficient. Looking forward, our teams are also working hard to bring the next generation of Zabbix features to life, with the following projects in the works:

Zabbix 8.0 LTS, which will introduce leading-edge technical features as well as a redesigned user interface with enhanced visualizations for a more intuitive and user-friendly experience.
Zabbix Mobile, an official Zabbix Mobile App for iOS and Android that will put instant push notifications, incident management, and seamless collaboration at your fingertips.
Zabbix Marketplace, a global platform designed to connect users, partners, and vendors in order to help them exchange information and discover new solutions together.

Growing our community

From major conferences to local meetups and knowledge-sharing events, the global Zabbix ecosystem came together like never before in 2025! Everything we do at Zabbix happens with the knowledge that our community is what sets us apart, which is why we’ll always meet our members wherever they happen to be.

Our global events gave monitoring professionals, partners, and enthusiasts a valuable forum to share their real-world expertise, helping our users keep pace with evolving technologies, strengthening their professional networks, and making sure that community feedback continues to shape our offerings. Some of the highlights included:

12 Zabbix Labs across Latin America, designed to foster vocational education and train specialized professionals who are ready for the professional challenges of the future.
1 regional forum in Mexico City that brought together regional experts to discuss trends, share experiences, and explore how open source tools are changing the technology landscape.
5 conferences (Benelux, Germany, China, Japan, and Latin America) that delivered a unique mix of practical knowledge, direct access to experts, and networking opportunities.
Innumerable exhibitions, trade fairs, and expos, which boosted our brand and built relationships.
One incredible Zabbix Summit in Riga that brought together hundreds of professionals, developers, partners, and users to share insights, learn from expert talks and workshops, and explore real-world case studies on effective Zabbix use.

Meanwhile, 2025 saw our headcount grow in every one of our global offices, as more and more talented individuals got wind of what we’re doing and chose to be a part of it.

Not to be outdone, our partners team also added 16 Resellers and 16 Certified Partners to our roster of global associates, all while revising and updating our Partner Program to bring Zabbix services to new users in more locations and in additional languages.

Staying safe

Security is not a one-time milestone for Zabbix, but a continuous process. In 2025, we focused on making our security practices more proactive, transparent, and deeply embedded into how we build and operate our products and services.

In 2025 we successfully recertified our ISO/IEC 27001:2022 and ISO/IEC 27017:2015 certifications for another 3 year period. At the same time, our HackerOne bug bounty program continued to be a solid line of defense, as 2025 brought 24 valid submissions that netted $16,700 in bounties.

Furthermore, as Latvia’s only CNA, Zabbix continued to refine its CVE workflows with faster triage and publication timelines, as well as improved vulnerability severity assessment and documentation.

Giving back

During the 2025 holiday season, we continued what may be the most important Zabbix tradition at all – lending our support to local organizations that make a real difference in our communities, including those that provided family support, food relief, access to healthcare and rehabilitation services, and simple holiday joy to senior citizens.

Due in large part to efforts like these, in November Zabbix was named “Enterprise of Riga 2025” in the ICT category – a recognition awarded to the most successful and impactful companies in Riga’s priority sectors. The award highlights not only business results but also contributions to innovation, sustainability, employee well-being, and the growth of the local tech ecosystem.

Looking ahead to 2026

In keeping with our new slogan “Your business works – you know it,” the plan for 2026 and beyond is to evolve from a traditional monitoring tool to a full observability platform, as our Founder and CEO laid out in his keynote speech during Zabbix Summit 2025.

We’re also planning to up our game when it comes to community engagement and training, meaning conferences, meetups, trainings, online events, and global expos that take into account feedback from our community and are increasingly tailored to provide members with what they need most.

With that in mind, we’d like to take this opportunity to thank our amazing global community for everything they did to make 2025 such a success. Whether it was in the form of blog contributions translating documentation, or creating templates and widgets, our community showed up in a big way all throughout the year.

It also goes without saying that we couldn’t have made 2025 the year that it was without our customers, whose trust, collaboration, and commitment to excellence continued to inspire us, drive innovation across everything we do, and allow us to stay open source and innovative.

“Looking ahead, I am genuinely excited about what is coming next in 2026. We have ambitious goals, but I have no doubt whatsoever that together, as one strong, amazing team, we will deliver, grow and continue building something we can all be proud of.” – Alexei Vladishev

The post Zabbix in 2025: A Year of Growth, Community, and Innovation appeared first on Zabbix Blog.

Keep Your Printers Happy with Zabbix and PaperCut NG

2025-12-23 Patrik Uytterhoeven

Post Syndicated from Patrik Uytterhoeven original https://blog.zabbix.com/keep-your-printers-happy-with-zabbix-and-papercut-ng/31705/

We all know the panic when the print system goes down. As I’ve written about before, PaperCut NG is a fantastic tool for managing printing, but even the best software needs a watchful eye to prevent unexpected downtime.

That’s why I’m excited to share a Zabbix template I developed that keeps a close, proactive check on your PaperCut environment. This isn’t about diving into complicated server logs, it’s about making your IT life easier by giving you clear, actionable alerts when printers start to go sideways.

The power of proactive monitoring

Why monitor your print server? It boils down to a few key points:

Stop Downtime Before It Starts: Imagine getting an alert that your database connection is shaky before users start complaining they can’t print. That’s the power of proactive monitoring.
Ensure Service Availability: PaperCut is critical for tracking costs, enforcing policies, and keeping things running smoothly. This template ensures the core service is always running smoothly.
Peace of Mind: Instead of manually checking system status pages, Zabbix becomes your automated, tireless assistant, ready to notify you instantly if there’s an issue.

What does the template monitor?

We have designed this template to focus on the key components that keep PaperCut NG running smoothly, using its built-in HTTP health checks to gather simple ‘yes/no’ answers about the system’s state.

Think of it as an automated checklist that runs every few minutes, reporting back on the most crucial parts of the service:

Application health: Is the main PaperCut service actually running and responding? (The most critical check!)
Database connectivity: PaperCut relies entirely on its database. We monitor to make sure the connection is solid and ready to log print jobs.
Printer status checks: We keep an eye on the printers themselves to ensure they are online and ready to accept print jobs, preventing user frustration from offline devices.

If any of these essential checks fail, Zabbix immediately raises a problem, allowing you or your team to jump in and fix the issue before the print queues fill up or staff can’t release their documents. Of course these are only some of the checks we have added.

Getting started is simpler than you think

You don’t need to be a Zabbix expert to start using this. The entire setup is focused on leveraging Zabbix’s powerful HTTP Agent capabilities, meaning you don’t need to install any extra software on your PaperCut server – just configure the right settings.

Here’s the high-level, non-technical process, fully detailed in the provided documentation:

Import the template: Download the template-papercut-http.yaml file and import it directly into your Zabbix server.
Add your PaperCut server: Create a new host in Zabbix representing your PaperCut server.
Link the template: Attach the newly imported PaperCut template to your host.
Configure access: The final step involves setting a simple, secure URL and a few configuration macros in Zabbix to tell the template where to check the PaperCut health status.

For step-by-step guidance on this process, you can refer to the full documentation: Monitoring PaperCut NG system health using Zabbix.

Try it out!

This template is open source and ready for you to implement, starting from Zabbix 7.0. It’s a great example of how simple, focused monitoring can save significant time and stress in a busy IT environment.

This project is a contribution from me, developed and made available through OpenSource ICT Solutions (OICTS). We believe in sharing simple, effective solutions to common IT challenges.

You can find the template and documentation on GitHub: OpensourceICTSolutions/ZabbixPapercutNG. Download it, test it, and let us know how it helps keep your printing infrastructure running smoothly!

If you need assistance with the migration or want to ensure best practices for scaling and optimizing Zabbix, don’t hesitate to reach out to OICTS. We are a Zabbix Premium Partner operating globally, with offices in the USA, UK, the Netherlands, and Belgium, and we’re ready to help you every step of the way.

The post Keep Your Printers Happy with Zabbix and PaperCut NG appeared first on Zabbix Blog.

Put Zabbix at your Fingertips with the IntelliTrend Mobile App

2025-12-16 Wolfgang Alper

Post Syndicated from Wolfgang Alper original https://blog.zabbix.com/put-zabbix-at-your-fingertips-with-the-intellitrend-mobile-app/31830/

The official Zabbix frontend works great on desktop, but it isn’t built for mobile. Monitoring doesn’t end when you step away from your workstation, and a reliable Zabbix mobile app keeps you connected to your Zabbix environment, gives you instant notifications, and allows you to react to problems or just check your host configuration at any time.

With IntelliTrend Mobile for Zabbix, you get a free, feature-rich mobile app for Zabbix, including real-time push notifications, custom mobile dashboards with unique widgets, built-in responsive host and item graphs, a detailed host viewer, and much more!

Mobile-optimized dashboards

Zabbix dashboards are great and powerful, but they are built for desktop screens and don’t always scale well on mobile devices. IntelliTrend Mobile solves that by giving you the ability to build as many dashboards as you need, each tailored to a specific purpose.

One dashboard can focus on infrastructure health, another on critical issues, and another on a single customer or environment. Every dashboard is an independent workspace, featuring its own layout, collection of widgets, and set of filter criteria.

Grid-based dashboard layout

Every IntelliTrend Mobile dashboard is powered by a grid-based layout system that gives you full control over how your dashboard looks and feels. You are not stuck with fixed widget sizes or a predefined structure – you can place widgets exactly where you want them, drag them around freely, and resize them to give each widget the space it really needs.

This grid system keeps everything orderly without boxing you in. Whether you build a clean, minimal dashboard or pack it with data-rich widgets, the editor helps you shape a layout that looks intentional and stays easy to work with.

Highly customizable widgets

The app offers a variety of unique and customizable widgets, each designed to display key monitoring information clearly and efficiently. Each widget comes with its own set of configuration options, so you can decide what it shows and how it shows it. You can filter widgets by hosts, host groups, severities, and much more in order to keep the view focused on what matters the most to you.

Besides that, widgets let you adjust their appearance by hiding or revealing extra details, switching between compact and extended modes or adjusting how much data they present. The result is a dashboard that is fine-tuned to the way you work.

Smart problem management

When issues happen, speed and context are everything. That’s why problem management is the most important part of any Zabbix mobile app. IntelliTrend Mobile is built to keep you informed the moment something goes wrong and to let you take action without wasting time or switching devices.

The problem view in the app gives you complete visibility into open and resolved issues, with filtering, sorting, and search options that let you quickly focus on the problems that require your attention. From the same interface, you can update and acknowledge problems without leaving the app.

Opening a problem takes you straight to a detailed view containing all the relevant information – severity, duration, related hosts, triggers, items, and historical data. What makes this view especially powerful is the ability to jump directly from the problem to the related host, item, or trigger within the app.

With a single tap, you can inspect the affected host, review item metrics, or analyze trigger history, all without leaving the mobile environment. This seamless navigation transforms problem management from a static list of alerts into a fully integrated, on-the-go investigation and resolution workflow.

Real-time response with Smart Alerts

The real game-changer, however, is IntelliTrend Mobile’s Smart Alerts feature. This isn’t just push notifications – it’s intelligent, actionable routing straight to the exact problem view in the app.

The moment a problem occurs, you’re notified in real time. Tap the alert and you’re immediately taken to the detailed problem screen. From there, you can analyze the issue, review metrics and history, acknowledge it, or take corrective action without ever opening the Zabbix web interface. No delays, no barriers, no switching devices!

With Smart Alerts, your team reacts faster, stays informed, and keeps systems running smoothly, turning mobile monitoring from passive alerts into active, on-the-go problem management.

Flexible problem list views

IntelliTrend Mobile lets you choose how problems are displayed in the list. By default, each problem appears as a detailed card showing all relevant information. If you prefer a cleaner overview, you can switch to a compact card view or even a compact list view, for maximum information density.

This flexibility is especially helpful when your Zabbix server generates many problems, allowing you to scan large numbers of problems at a glance while keeping the interface tidy and manageable.

View item and host graphs with mobile-optimized charts

IntelliTrend Mobile reshapes the way you view Zabbix data while you’re on the move. Instead of relying on Zabbix’s static, desktop-focused graphs, the app renders item histories, host graphs, service uptimes, and SLA metrics using fully native, mobile-friendly charts. These charts are responsive, adapting seamlessly to your screen size and orientation for a smooth and clear viewing experience, whether you’re on a phone or tablet.

Every graph is interactive. You can zoom in to inspect a specific time window, pan across the timeline, or hover with your finger to see precise data points. Multiple data series can be toggled on or off, making it easy to focus on the metrics that matter.

You can quickly switch between time periods and pinpoint when an issue started, track its progression, or confirm when it was resolved – without ever opening the Zabbix web interface.

Complete host overview

You can also view all the essential details about any host right from the mobile app. Every host has a detailed view that puts all relevant information at your fingertips, making management simple, efficient, and fully mobile.

For each host, you can quickly see its visible and technical names, current status (enabled or disabled), and maintenance state, including whether data collection continues during maintenance. If the host is monitored by a proxy, you see the proxy that monitors it.

The host details view gives you instant access to all related configurations and objects:

Templates and host groups

You can view all templates assigned to a host and dive into any template’s full details with a single tap, making it easy to understand the monitoring configuration at a glance. Host groups work the same way – just tap a group to see every host it contains, giving you instant insight into related systems.

Host interfaces

IntelliTrend Mobile gives you a complete view of each host interface, including agent, SNMP, JMX, and IPMI types. For every interface, you can see its IP address, DNS name, port, and the interface type configured in Zabbix.

The app also shows the current availability status, highlighting interfaces that are unreachable or experiencing errors. This makes it easy to quickly identify connectivity problems, verify which monitoring protocols are active, and troubleshoot issues with data collection.

Macros

Macros are displayed with full detail (including type, value, and description) so you can verify configuration settings quickly or troubleshoot dynamically, all without leaving the host view.

Inventory

The host inventory view in IntelliTrend Mobile gives you full access to the complete Zabbix host inventory. All inventory fields configured in Zabbix are displayed directly in the app, giving you a complete overview of the host’s recorded details. You can also see the inventory mode for the host (Disabled, Manual, or Automatic) so it’s immediately clear how the inventory is being managed.

Open and resolved problems

From the host details page, you can jump straight into all open or resolved problems related to that specific host. One tap takes you directly to a filtered problem list, making it effortless to review recent problems or check the current state of the host without navigating through multiple menus.

Items, triggers, and graphs

All items, triggers, and graphs tied to the host are just one step away. Each entry opens a filtered list focused solely on that host, letting you move from the host view into any related object instantly. Whether you need to inspect a value, review a trigger, or explore a graph, the app keeps the entire chain of information connected.

Scripts

Execute host scripts directly from your mobile device, whether you’re restarting a service, collecting diagnostics, or triggering an automated workflow. It’s a fast, practical way to take action remotely, enabling real operations work even when you’re away from your desk.

With all these features combined, the host details view becomes a powerful, fully mobile workflow. Everything you need to monitor, analyze, and take action is right at your fingertips, making host management faster, more efficient, and truly on-the-go.

Customize your views

Favorites

You can create favorites for specific hosts or host groups and quickly switch your global scope to focus on them. Once a favorite is active, the app automatically filters all dashboards and list pages to show only data related to that host or host group.

Favorites make it easier to concentrate on the systems you manage most often, so you don’t have to reapply filters or navigate through long lists every time. You can switch between favorites at any time, giving you a fast way to move between different parts of your environment.

Layout modes

Everyone works differently, so the app comes with useful customization options. In addition to the filtering and sorting available on every list page, you can switch between different layout modes for all list pages.

Choose from the standard layout, a compact card layout, or a compact list layout for maximum information density. This lets you decide how much information you want to see at once and allows you to apply layout preferences individually for each page or set them globally in the app settings.

Many more features

The app already supports a wide range of features designed to give you full visibility into your Zabbix environment. Beyond the previously mentioned features, the app includes many more features, such as accessing services and SLAs to keep track of service performance and availability, explore templates, and view triggers, items, and graphs in detail.

But our development doesn’t stop here. We are constantly expanding app functionality and improving existing features based on user feedback. If you haven’t tried the app yet, now is a great time! We’d love to hear your honest thoughts about what works well, what could be better, and which features you’d like to see next. Your feedback helps to shape the future of IntelliTrend Mobile, and we take every suggestion seriously.

Submit feedback

The post Put Zabbix at your Fingertips with the IntelliTrend Mobile App appeared first on Zabbix Blog.

24/7 Alerting and Two-Way Integration with Zabbix and SIGNL4

2025-12-09 Ronald Czachara

Post Syndicated from Ronald Czachara original https://blog.zabbix.com/24-7-alerting-and-two-way-integration-with-zabbix-and-signl4/31866/

It’s a familiar story for many IT operations teams: a critical server went down overnight, but the alert was buried in someone’s inbox. By the time anyone noticed, valuable time was lost, SLAs were breached, and the team spent the next morning explaining why an email hadn’t been seen. Email (or even SMS text) alone simply wasn’t reliable enough for something as urgent as incident alerts.

The turning point came when the team decided to integrate SIGNL4 with Zabbix. Setup was fast – within minutes, alerts that once hid in crowded inboxes were now reaching the right on-call engineer – loud, clear, and actionable. Instead of reacting late, the team was responding in real time and the night shifts suddenly felt a lot less stressful.

Integration overview and two-way communication

The SIGNL4 integration leverages a Zabbix media type to seamlessly send event data from Zabbix to SIGNL4. Once configured, Zabbix alerts are instantly transformed into mobile push notifications, ensuring rapid delivery and clear visibility for on-call teams.

Beyond alerting, the integration also supports bidirectional status updates between the two systems – including acknowledgements, closures, and annotations. When an on-call engineer acknowledges an alert in the SIGNL4 mobile app, the status is automatically reflected in the Zabbix dashboard.

Likewise, when Zabbix detects recovery (status UP), it triggers an automatic update to close the corresponding alert in SIGNL4. This real-time synchronization keeps both platforms perfectly aligned, maintaining consistent alert and recovery states without any manual effort.

Configuration steps

In the Zabbix web portal go to “Alerts” -> “Media types.”

Find the SIGNL4 media type, enable it, and enter your SIGNL4 team or integration secret in the parameter “teamsecret.” Alternatively, you can leave the default ({ALERT.SENDTO}) and enter the SIGNL4 team secret into the “Send to” parameter of your user.

Update the settings:

In the media type list click the button “Test” for the SIGNL4 media type to send a test alert. You will receive an alert in your SIGNL4 mobile app.

Under “User settings” -> “Profile” go to “Media” and add the SIGNL4 media type here. Adapt the alerting settings according to your needs.

That’s it! Now a SIGNL4 alert is triggered every time Zabbix sends an alert to your Zabbix user.

Back-channel configuration for status updates

In the SIGNL4 web portal go to “Integrations” -> “Gallery” and look for the “Zabbix ()” integration. Note the arrow pointing to the left.

As “Zabbix URL” enter your Zabbix URL, e.g. https://your-zabbix-server/

Next, enter “Your Zabbix API token.” You can find this one as described here.

There’s no need for a username and password – just use the API token.

Enable the integration and click “Install.”

That’s it! Status updates are now sent from SIGNL4 to Zabbix.

For more information, have a look at the integration guide.

Key benefits

24/7 Alerting and escalation – Critical Zabbix alerts reach the right people instantly via mobile app, push, SMS, or voice call. This includes escalation, ensuring nothing slips through the cracks.
On-call duty management – Calendar-based on-call scheduling and automated routing replaces manual escalation, helping teams sleep better and respond smarter.
Rich, mobile-first notifications – Alerts include key incident details, so engineers can act quickly without logging into dashboards first.
Team collaboration and acknowledgment tracking – Everyone sees who has picked up an alert, for full transparency and structures response.
Reduced MTTA/MTTR – Faster acknowledgment and resolution mean less downtime, fewer escalations, and more stable operations.

What once felt like a constant struggle with missed notifications has turned into a structured, reliable alerting process. By connecting Zabbix with SIGNL4, the team not only strengthened their incident response but also made on-call duty a lot less of a burden – and that might be the biggest win of all.

The post 24/7 Alerting and Two-Way Integration with Zabbix and SIGNL4 appeared first on Zabbix Blog.

Saving Time with a Custom Zabbix Agent Installer

2025-12-04 Rizqi Firmansyah

Post Syndicated from Rizqi Firmansyah original https://blog.zabbix.com/saving-time-with-a-custom-zabbix-agent-installer/31843/

When managing large-scale infrastructure, the process of installing monitoring agents is often repetitive and time-consuming. Administrators must log into each server, manually run installation commands, and configure the agent to connect to the Zabbix server. To address this issue, the Zabbix Agent Deployer custom module was created. This module enables the direct installation of Zabbix agents on multiple hosts from the Zabbix Web interface.

The features of the Zabbix Agent Deployer module include:

Bulk host list input using a CSV file.
The ability to automatically add hosts to Zabbix and remotely install the Zabbix Agent on the
associated hosts.
The ability to display installation log results directly within the module.

With this approach, administrators can add new hosts to the monitoring system faster and more efficiently.

Key use cases for the Zabbix Agent installer

The Zabbix Agent Deployer module enables several practical scenarios, including:

1. Faster provisioning for new servers – When adding a large number of servers, agents can be installed simultaneously without requiring a login to each machine.

2. Standardized installation – All agents are installed in the same way using a centralized script, reducing the risk of misconfiguration.

3. Easier additional provisioning – Provisioning new servers is easier for users because they don’t need to configure them directly on the server.

Getting started with the Zabbix Agent Deployer module

Solution overview architecture

To use this module, the main steps are:

1. Upload the custom module to the Zabbix frontend in the /usr/share/zabbix/modules/ directory.

2. Enable the module from the Administration → General → Modules page, and click the Scan Directory button. Locate the Zabbix agent deployer module and click Enabled.

3. Once activated, the Zabbix agent deployer module can be accessed in the Data Collection menu. Here’s a screenshot of the Zabbix agent deployer module.

4. Prepare a CSV file like the format below, or download a sample CSV from the module page.

With this CSV file, we will add two hosts to Zabbix to be monitored and automatically install the Zabbix agent on them.

5. Upload the CSV file to the Zabbix agent deployer module page and click Apply.

6. The Zabbix agent deployer module will handle the process of adding hosts to Zabbix and installing the Zabbix agent. The status can be seen as follows:

From the image above, server1 and server2 were successfully added to Zabbix, and the Zabbix agent installation was successful!

7. Check out the Zabbix hosts list page. Hosts will appear according to the uploaded CSV file.

Conclusion

The implementation of this custom Zabbix Agent installer extends Zabbix’s capabilities beyond its built-in functionality. The Zabbix Agent Deployer module enables a more efficient bulk host addition process, as all steps from adding hosts to Zabbix to installing the Zabbix agent can be integrated through a single page.

If you’re interested in implementing this, please contact us. Bangunindo is a premium Zabbix partner in Indonesia. We’re ready to help you design, implement, and optimize your Zabbix solution to suit your needs.

The post Saving Time with a Custom Zabbix Agent Installer appeared first on Zabbix Blog.

Aruba Central API Monitoring with Zabbix

2025-11-25 Tibor Volanszki

Post Syndicated from Tibor Volanszki original https://blog.zabbix.com/aruba-central-api-monitoring-with-zabbix/31370/

Aruba Central is a SaaS solution that allows you to manage your Enterprise Aruba network environment. Due to the increasing number of cloud migrations, we can expect that more and more Aruba customers will move their on-premise environment to it, which will also mean a change in their monitoring environment. In this article, I will show you how to switch to API- based monitoring using Aruba Central and Zabbix. All custom resources mentioned can be found in my repository.

Aruba Central’s API

Oauth 2.0 is used, so you can forget the simple token management. At the end it is great, but for monitoring purposes it is overkill. There is pretty good documentation (referred to later) regarding how you can generate your access token, but after two hours it expires so you need to continually refresh it. To do this, you must use a refresh token, which can help you to get a new access token AND a new refresh token.

Within two hours, use the latest refresh token to repeat this action again. At this point you can imagine that this is not something you can implement easily by using the Zabbix GUI only. Well, maybe with some javascript magic, but otherwise there is no native support for this logic at this point of time. So how can we do this? In short:

Generate your client credentials
Generate your first token
Schedule the token refresh for every two hours
Update your host macro via Zabbix API
Use the token in Zabbix HTTP agent checks
Monitor your environment based on JSONPath pre-processing

Initial steps within Aruba Central

To manage your API access, you need to launch your “HPE Aruba Networking Central” application, so do NOT look into your workspace modules – the “Personal API clients” menu is NOT what we are looking for. Turn off the “New Central” view – at this point the early access version is not so useful (hopefully it will change soon).

The first time you get there, you will not see any items, but under the “My Apps & Tokens” tab you can click the “Add Apps & Tokens” button and generate it. Technically, this is already enough to start to monitoring your network infrastructure, but within two hours it would stop. So the relevant data for us are the “Client ID” and “Client Secret.” Feel free to revoke the recently created token at the bottom area as we do not need it.

Record your credentials

For this article, I am using a simple file to store all the credentials, which will be sourced into a bash script. Please keep in mind that storing your sensitive credentials in a single file is a BAD practice! Your SECO/CISO would probably have a few words with you about it, so please consider a better approach. A more secure way would be to use some Key Vault solution (like Azure, AWS, Google, or Hashicorp). Anyway, let’s continue with this unsecure example:

#!/bin/bash

### ZABBIX VARS ###

# URL of your zabbix instance (assuming you do not use the "/zabbix" ending, if yes, then add it to the end)
zabbix_url="https://your.zabbix.instance.net"
# Your Zabbix API token. If you do not know how to get it, check the documentation.
zabbix_api_token="1234_your_zabbix_api_key_5678"
# Create a host with a macro, remain at the "Macros" tab, turn on debug mode, look for "[hostmacroid] =>"
zabbix_macro_id="12345"

### ARUBA VARS ###
# To find yours, go here and check "Table: Domain URLs for API Gateway Access"
base_url="YOUR_ARUBA_CENTRAL_BASE_URL"
# Click on your profile in the Central app and you will find it there: 32 char long hexa string
client_id="YOUR_CLIENT_ID"
# provided in the previous step
client_secret="YOUR_CLIENT_ID"
# provided in the previous step
customer_id="YOUR_CUSTOMER_ID"
# your login credential
account_username="YOUR_CENTRAL_LOGIN_USERNAME"
# your login credential
account_password="YOUR_CENTRAL_LOGIN_PASSWORD"
# to be populated later
csrftoken=""
session=""
auth_code=""

Get or refresh your token and update the Zabbix host macro

The next steps are based on the official Aruba documentation, which you can find here. Please remember that there are many ways to achieve our target – this is just one example and probably not the most optimal one. Feel free to change / improve it with your code in your preferred scripting language.

The below script assumes that the file containing the credentials (previous step) is named as “variables” and located in the folder named “central.”

Filename: aruba_central_token_new.sh

Purpose: To be used for first time token generation. Later, you only have to refresh your token with the script after this one.

Remarks: Aruba is limiting this API query set, so you can run it only ONCE every 30 minutes! If you made a typo somewhere, wait 30 minutes before your next attempt or tweak the result files.

#!/bin/bash

basedir=central
source $basedir/variables

curl -s --noproxy '*' -v --cookie-jar $basedir/cookie --location --request POST "$base_url/oauth2/authorize/central/api/login?client_id=$client_id" \
--header "Content-Type: application/json" \
--data-raw "{
    \"username\": \"$account_username\",
    \"password\": \"$account_password\"
}" > $basedir/result1.raw 2>&1

grep 'Added cookie' $basedir/result1.raw > $basedir/result1.filtered

csrftoken=$(grep csrftoken $basedir/result1.filtered | awk -F '"' '{print $2}')
session=$(grep session $basedir/result1.filtered | awk -F '"' '{print $2}')

curl -s --noproxy '*' --request POST "$base_url/oauth2/authorize/central/api?client_id=$client_id&response_type=code&scope=all" \
--header "Content-Type: application/json" \
--header "Cookie: session=$session" \
--header "X-CSRF-Token: $csrftoken" \
--data-raw "{
\"customer_id\": \"$customer_id\"
}" > $basedir/result2.raw

auth_code=$(cat $basedir/result2.raw | jq -r .auth_code)

curl -s --noproxy '*' --request POST "$base_url/oauth2/token" \
--header "Content-Type: application/json" \
--data "{
    \"client_id\": \"${client_id}\",
    \"client_secret\": \"${client_secret}\",
    \"grant_type\": \"authorization_code\",
    \"code\": \"${auth_code}\"         
}" > $basedir/result3.raw

refresh_token=$(cat $basedir/result3.raw | jq -r .refresh_token)
access_token=$(cat $basedir/result3.raw | jq -r .access_token)

if [ "$refresh_token" == "null" ]; then
    echo "something went wrong... exiting now"
    exit 1
fi

echo $access_token > $basedir/token_access.latest
echo $refresh_token > $basedir/token_refresh.latest

echo "access_token: $access_token"
echo "refresh_token: $refresh_token"

curl -s --request POST \
--url "$zabbix_url/api_jsonrpc.php" \
--header "Authorization: Bearer $zabbix_api_token" \
--header "Content-Type: application/json-rpc" \
--data "{\"jsonrpc\": \"2.0\",\"method\": \"usermacro.update\",\"params\": {\"hostmacroid\": \"${zabbix_macro_id}\",\"value\": \"${access_token_new}\"},\"id\": 1}"

rm -f $basedir/cookie

Filename: aruba_central_token_refresh.sh

Purpose: To refresh your existing token. It is expecting an existing refresh token in the “token_refresh.latest” file, so better to run the previous script one time before this.

Remarks: You can run this script as many times you want, but it will result in new tokens only once per every two hours (when the current one expires). Therefore, refreshing too frequently is pointless.

#!/bin/bash

basedir=central
source $basedir/variables

refresh_token_current=$(cat $basedir/token_refresh.latest | tr -d '\n')
refresh_token_new=""

curl -s --noproxy '*' --request POST "$base_url/oauth2/token?client_id=$client_id&client_secret=$client_secret&grant_type=refresh_token&refresh_token=$refresh_token_current" > $basedir/result4.raw

refresh_token_new=$(cat $basedir/result4.raw | jq -r .refresh_token)
access_token_new=$(cat $basedir/result4.raw | jq -r .access_token)
expires_in=$(cat $basedir/result4.raw | jq -r .expires_in)

if [ "$refresh_token_new" == "null" ]; then
    echo "something went wrong... exiting now"
    exit 1
fi

echo $access_token_new > $basedir/token_access.latest
echo $refresh_token_new > $basedir/token_refresh.latest

echo "access_token: $access_token_new"
echo "refresh_token: $refresh_token_new"
echo "expires_in: $expires_in"

curl -s --request POST \
--url "$zabbix_url/api_jsonrpc.php" \
--header "Authorization: Bearer $zabbix_api_token" \
--header "Content-Type: application/json-rpc" \
--data "{\"jsonrpc\": \"2.0\",\"method\": \"usermacro.update\",\"params\": {\"hostmacroid\": \"${zabbix_macro_id}\",\"value\": \"${access_token_new}\"},\"id\": 1}"

In my case, both the scripts and variables files are in the same “central” folder, which is in a git repository. Each time I call one of the scripts, it will record the new tokens in files, which are committed and pushed to the repo. In my own implementation, this is how I call the refresh script and sync the result with my repo:

git checkout master

basedir=central
source $basedir/variables
bash $basedir/aruba_central_token_refresh.sh

git add .
git commit -m "save the new tokens"
git push origin master

Schedule your token management

You must run your refresh script at least once per every two hours. To make this happen you have many options, including:

cron (old-school, outdated way)
systemctl timer (a better way, but only if it is monitored)
Jenkins / Github Actions/etc.
Zabbix itself, by calling your bash script

In my case, Jenkins does the scheduling and execution and the job is monitored via Zabbix.

Monitor your network infrastructure

When everything is in place, then the monitoring part is pretty simple. The usual JSONPath based logic can be used. API call documentation can be found here. The template contains only the wireless components, since I do not have my switches in Central. Implementing the switching part should not be difficult – just have a look at the “Switch” section, then clone and adjust one of your “get” items.

Screenshots

Latest data – tag based filtering:

Latest data – Site health

Latest data – Gateway info

Latest data – AP info

Triggers:

Some triggers are intentionally disabled, because they are a bit redundant. However, I wanted to cover all options. Sometimes less alerting is better if you have a ticketing system integration, otherwise your monitoring system will turn into a ticket factory.

Known issues and limitations

Since we are not querying the devices directly, some delay can be expected. Based on my recent testing, the delay compared to real time is between 3-10 minutes. In my test I disconnected my test environment and then started to do manual updates frequently. Some items got the real state earlier, some only later.

If your refresh script will malfunction for whatever reason (normally it should not), then you may have to run the other script once to generate a new token, or you can go to the GUI and check the last refresh token, with which you can override the content of the “token_refresh.latest” file.

Aruba is limiting the number of API queries to 5,000 per day. This could seem annoying, but it is way more than what you need (you should expect less than 1,000 in normal conditions, depending on your update frequency).

Zabbix API will not authorize your call unless you insert a line into your apache vhost configuration. This is a more generic Zabbix API issue that is not related to Aruba Central.

SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1

If Aruba Central has a maintenance activity, then the token refreshing way could break. Running the token request script once should address the issue.

Summary

Aruba Central’s API is pretty decent, but if you start from zero it could take a while to get to the end of it. With this guide, my intention was to speed you up, but please do not consider my scripts and the shown example as the only or best possible way – I’m just hoping it can give you a good base for your own solution. Have fun!

The post Aruba Central API Monitoring with Zabbix appeared first on Zabbix Blog.

Optimize latency-sensitive workloads with Amazon EC2 detailed NVMe statistics

2025-11-19 Sanjeev Malladi

Post Syndicated from Sanjeev Malladi original https://aws.amazon.com/blogs/compute/optimize-latency-sensitive-workloads-with-amazon-ec2-detailed-nvme-statistics/

Amazon Elastic Cloud Compute (Amazon EC2) instances with locally attached NVMe storage can provide the performance needed for workloads demanding ultra-low latency and high I/O throughput. High-performance workloads, from high-frequency trading applications and in-memory databases to real-time analytics engines and AI/ML inference, need comprehensive performance tracking. Operating system tools like iostat and sar provide valuable system-level insights, and Amazon CloudWatch offers important disk IOPs and throughput measurements, but high-performance workloads can benefit from even more detailed visibility into instance store performance.

For latency-sensitive applications where every millisecond counts, enhanced performance monitoring tools provide deep visibility into storage systems, so your teams can track and analyze behavior at a 1 second granularity. This detailed insight can help your organization detect bottlenecks quickly, fine-tune application performance, and deliver reliable service.

In this post, we discuss how you can use Amazon EC2 detailed performance statistics for instance store NVMe volumes, a set of new metrics that provide per-second granularity, to provide real-time visibility into your locally attached storage performance. These statistics are similar to the Amazon EBS detailed performance statistics, providing a consistent monitoring experience across both storage types. You can access these statistics directly from your NVMe devices attached to the Amazon EC2 instance using nvme-cli or using CloudWatch agent to monitor I/O performance at the storage level. We also provide examples of how to use these statistics to identify performance bottlenecks.

Feature overview

Amazon EC2 Nitro-based instances with locally attached NVMe instance storage now offer 11 comprehensive metrics at per-second granularity. These metrics, similar to EBS volume metrics, include queue length measurements, IOPS, throughput data, and IO latency histograms for the locally attached NVMe instance storage. Additionally, they also include IO size-specific latency histograms to provide even more detailed insights into performance patterns of the local NVMe instance storage. These metrics are collected and presented separately for each individual NVMe volume available on an instance.

The statistics are presented in three main formats:

1. Cumulative counters that track IO operations, throughput, and read/write times
2. Real-time queue length, displaying the current value at the time of your query
3. Latency histograms visualizing the distribution of IO operations across different latency ranges by displaying both cumulative view and IO size-specific distributions

Prerequisites

To access detailed performance statistics for local instance storage, complete the following steps:

1. Launch a new Amazon EC2 Nitro instance or use an existing one, then connect to it using SSH or your preferred connection method.
2. Identify the NVMe device associated with the local storage to query for the performance statistics. For example, you can run the nvme-cli command in the CLI to output all NVMe devices on the instance.
```
$ sudo nvme list.
```
  The following is an example output of the list command that lists the NVMe devices on the instance and their volume Serial Numbers (SN; masked in the below output for privacy). In this demonstration, consider that the local storage used by your application is /dev/nvme1n1.
3. If you are using Amazon Linux 2023 version 2023.8.20250915 (or later) or Amazon Linux 2 2.0.20251014.0 (or later) you can proceed to Step 4 because nvme-cli will use the latest version. If you are using an earlier Amazon Linux version, update the nvme-cli using the following command, where 2023.8.20250915 can be replaced with the latest Amazon Linux 2023 version:
```
$ sudo dnf upgrade --releasever=2023.8.20250915
```
4. Run the nvme-cli, with the correct permissions, and pass the device as a parameter. You can use --help to get details on the command usage:
```
$ sudo nvme amzn stats --help
```
  Example output:
  
  If you prefer output in a JSON format, you can provide the -o json parameter to the command.
```
$ sudo nvme amzn stats /dev/nvme1n1 -o json
```
  The following output (without the -o json parameter) shows cumulative read/write operations, read/write bytes, total processing time (read and write in microseconds), and duration (in microseconds) when application attempted to exceed the instance’s IOPS/throughput limits.
  
  It also displays read/write I/O latency histograms, with each row representing completed I/O operations within a specific bin of time (in microseconds).
  
  If you want to view the latency histograms across 5 different IO bands: (0, 512 Byte], (512B, 4KiB], (4KiB, 8KiB], (8KiB 32KiB], (32 KiB, MAX], you can provide --details or -d parameter to the command:
```
$ sudo nvme amzn stats -d /dev/nvme1n
```
  The following image is an excerpt of the above command’s output, showing the additional latency histograms (read and write) of the 5 different IO bands.

You can run the stats command at a per second granularity. You can also write scripts to pull the stats at a desired interval (every second or any other duration) with each subsequent output reflecting the updated cumulative totals for the metrics. Calculating the difference in the statistics across the last two outputs allows you to derive insight into the instance storage profile during the interval. Below is a sample script you can use to pull the stats at a default interval of 1 second or at your desired interval.

#!/bin/bash 
# interval of 1 second 
INTERVAL=${1:-1} 
while true; do 
	echo "=== $(date) ===" 
	sudo nvme amzn stats /dev/nvme1 || break 
	echo 
	sleep $INTERVAL 
done

You can save this script, make it executable and run it at either the default 1-second interval or provide a custom interval when executing the script. For example, if you saved the script as nvme_stats.sh, you could use the following commands to make it executable and run to get the output at the default 1-second interval (assuming you are in the same directory as that of nvme_stats.sh).

chmod +x nvme_stats.sh
./nvme_stats.sh

If, for instance, you want to get the output at every 5 seconds, you can use the command below (after making the script executable)

./nvme_stats.sh 5

You can also integrate with CloudWatch using CloudWatch agent to collect and publish these statistics for historical tracking, trend visualization through dashboards, and performance-based alerts to correlate with application metrics and automated notifications for performance issues.

Deriving insights from the Amazon EC2 instance store NVMe detailed performance statistics

Similar to EBS detailed performance statistics, you can use Amazon EC2 instance store NVMe statistics to troubleshoot various workload performance issues. As mentioned in the preceding section, you can also use the detailed statistics to view I/O latency histograms to observe the spread of I/O latency within the period. You can use the read/write operations and time spent statistics to calculate the average latency. The detailed statistics show the average latency at per-second granularity.

The next two example scenarios demonstrate key performance analysis using the statistics. In Scenario 1, we will use the EC2 Instance Local Storage Performance Exceeded (us) metric to check if I/O demands exceed instance storage capabilities, helping with instance right-sizing for sufficient I/O application performance. In Scenario 2, we will use IO-size specific histograms (using --details) to diagnose how large block writes affect subsequent read performance – an issue typically hidden by traditional monitoring tools’ aggregated metrics across all IO sizes.

Scenario 1: Identifying when applications exceed instance storage performance limits

Understanding whether your application’s I/O demands exceed your instance store volumes’ capabilities is important for performance troubleshooting. When applications generate I/O workloads that consistently attempt to exceed the IOPS and throughput limits of specific Amazon EC2 instance types, you’ll experience increased latency and degraded performance. The EC2 Instance Local Storage Performance Exceeded (us) metric helps identify these scenarios by showing the duration (in microseconds) when workloads exceeded supported instance performance. A non-zero value or increasing count between snapshots indicates your current instance size or type may not provide sufficient I/O performance for your application.

The following section shows how to identify if an application is sending more IOPS than the instance’s local storage can support.

The example scenario: An application on an i3en.xlarge instance shows elevated write latency of >1ms. You want to determine if the application’s workload is exceeding the instance’s NVMe volume supported performance.

1. Select the Instance Storage NVMe device you want to analyze – Identify the instance you want to analyze for the application experiencing elevated latency.
2. Identify the NVMe device – Use the following nvme-cli command, and identify the NVMe device associated with that instance storage.
```
$ sudo nvme list
```
  Example scenario: We used the list and identified /dev/nvme1n1 as the NVMe device associated with the i3en.xlarge instance that is running the application which is currently seeing elevated write latency >1ms (while read latency is <50us as per normal conditions), so now we want to. analyze it.
3. Collect statistics for the device at a single point in time or at desired intervals – Collect the detailed performance statistics using the nvme-cli command or use the sample script provided in previous section to capture statistics at the desired intervals, if needed.
```
$ sudo nvme amzn stats /dev/nvme1n1
```
  Example scenario: We choose to collect the statistics only once after noticing elevated write latency of the application.
4. Analyze the statistics to check if the application demands more than the supported performance of the instance storage – Confirm existence of overall I/O latency degradation by comparing two sets of read/write I/O latency histograms taken some time apart.Example scenario: The following output shows Read IO histogram of the NVMe local instance storage taken 40 seconds apart with no read IO latency issues (as normal read latency for this workload is < 50 us).
  Metric captured at time T:
  
  Metric captured at time T+40s:
  
  The following output shows Write IO histogram taken 40 seconds apart. We can discern that many write IOs fall into the 1ms – 2ms latency range, which is not expected for this application.
  Metric captured at time T:
  
  Metric captured at time T+40s:
5. Analyze the EC2 Instance Local Storage Performance Exceeded (us) metric which shows total time (in microseconds) IOPS requests exceed volume limits. Ideally, the incremental count of this metric between two snapshot times should be minimal, as any value above 0 indicates that the workload demanded more IOPS than the volume could deliver.Example scenario: Comparing metrics 40 seconds apart shows that for more than 34 seconds, the application’s IOPS demands surpassed the IOPS supported by the local instance storage. This explains elevated write latency: excess IOPS above what the underlying storage can physically handle queue the operations, increasing wait times. This indicates that the i3en.xlarge instance chosen to run this application cannot meet the application’s performance requirements, suggesting either upgrading to a larger instance size or re-evaluating the instance type itself.
  Metric captured at time T:
  
  Metric captured at time T+40s:

It’s important to have the right instance size to avoid performance bottlenecks to your application. Refer to the Amazon EC2 instance documentation for more information on the different instances and their storage size.

Scenario 2: Identifying the block size causing elevated latency in your applications

Many storage performance issues arise from complex interactions between read and write operations with different I/O sizes, which traditional system-level monitoring tools like iostat or sar cannot effectively diagnose due to their aggregated metrics across all I/O sizes. EC2 instance store NVMe detailed performance statistics solves this by providing I/O-size specific latency histograms through the --details option in NVMe CLI. These histograms show latency data for different I/O size ranges: (0, 512 Byte], (512B, 4KiB], (4KiB, 8KiB], (8KiB, 32KiB], (32KiB, MAX], for a more precise correlation between application workload patterns and I/O size-specific latency metrics for targeted optimizations.

In this example scenario, your application performs small reads (typically <=4KiB, like metadata read) followed by large writes (>=32KiB) and shows unexpectedly high read latency. This common issue occurs when large writes impact subsequent read operations’ performance, creating a cascading effect on overall I/O performance.

1. Gather read and write IO latency by size ranges – Use the NVMe CLI with the --details option to gather read and write IO latency by size ranges:
```
$ sudo nvme amzn stats /dev/nvme1n1 --details
```
2. Confirm existence of overall IO latency degradation – In the example scenario, examining overall IO latency, both read (left) and write (right) operations are showing higher than expected latency.
3. Examine the output for patterns across different IO size bands – Analyzing latency by operation sizes shows small read operations (512 bytes to 4K), typically fast, are experiencing unexpected latency spikes while large writes (32K+) show significant delays. Small reads should theoretically maintain good performance regardless of other I/O activities.
  
  The observed pattern indicates that the backed-up large write operations create system-wide congestion, affecting all I/O operations of types and sizes. Despite the storage system’s capability to handle small reads efficiently, the queued large writes slow down both read and write operations at the application level.

Based on this analysis, we can implement several targeted optimizations to the application, like using smaller block sizes for write operations when possible, or batching smaller writes instead of performing large single writes.

Clean up

If you created an Amazon EC2 instance with NVMe volume for this exercise, then terminate and delete the appropriate instance to avoid future costs.

Conclusion

Amazon EC2 detailed performance statistics for instance store NVMe volumes provide real-time, sub-minute storage performance monitoring, similar to the detailed performance statistics available for Amazon EBS volumes. This offers consistent monitoring experience across both storage types, with additional IO-size based latency histograms for instance storage for better optimization of I/O patterns, and more effective troubleshooting.

To learn more about Amazon EC2 instance store NVMe volumes, optimization techniques for latency-sensitive workloads or other Amazon EC2 related topics, visit the Amazon EC2 documentation page or explore our other AWS Storage Blog posts on performance optimization.

We’d love to hear how you’re using these statistics to enhance your workloads, or if you have any questions, in the comments section below.

Improving Customer Satisfaction and Experience with Zabbix

2025-11-07 Michael Kammer

Post Syndicated from Michael Kammer original https://blog.zabbix.com/improving-customer-satisfaction-and-experience-with-zabbix/31692/

No matter what business you’re in, there is one universal truth – your success or failure depends on customer satisfaction and trust. And when your IT systems fail, it’s your customers who pay the price. Being unable to place an order due to unexpected downtime (which can cost a large organization as much as $9,000 per minute) or having their credit card data compromised in a preventable security breach (which costs the average organization nearly $5 million) will force even your most loyal customers to go somewhere else.

Monitoring with Zabbix doesn’t just keep your infrastructure safe, it keeps your reputation safe and makes sure that your customers continue to be your customers. It does this by guaranteeing the performance, reliability, and security of your digital services – while also supporting better customer service and continuous improvement. Keep reading to see how it’s possible.

Table of Contents

Say goodbye to downtime

Your customers are looking to meet their needs quickly and effectively. Unexpected service disruptions cause them to feel neglected and force them to look elsewhere for solutions.
Monitoring your infrastructure with Zabbix can effectively eliminate downtime through proactive issue detection, which locates anomalies and performance issues like high CPU usage, packet loss, and latency in real time – before they have a chance to make life harder for customers.

If an issue does occur, Zabbix’s predictive alerting capabilities let your tech teams know about anything that could potentially impact an application or service, which lets them meet SLAs and provide a better, more reliable customer experience with fewer service disruptions, which in turns leads to higher levels of trust and satisfaction.

Outperform your competitors

No matter how good your products or services happen to be, you still need to provide smooth and fast online user experience if you want repeat use and positive reviews. Monitoring with Zabbix optimizes network traffic by helping you to identify bandwidth bottlenecks or misconfigured devices with a single glance at a dashboard, allowing better traffic management and a better online experience for customers.

It also improves response times, which allows you to be confident that your applications and services remain responsive. This is especially important for real-time services like video conferencing, e-commerce, or customer support.

Turn good customer service into outstanding customer service

What turns a casual, one-time user into a repeat customer? In most cases, it all comes down to making that user feel seen, informed, and supported. Zabbix helps you maintain consistent system performance, and nothing builds trust like stability.

With a bit of configuration and the help of IT service management tools like ServiceNow, Zabbix can provide clear, easy-to-access logs and metrics that help your customer service reps better understand your customers and the process of serving them, including:

• Customer satisfaction (CSAT)
• Preferred communication channel
• Average ticket count
• Average response time
• Average ticket resolution time
• Ticket resolution rate
• Ticket backlog
• Interactions per ticket

With this information, your team will be able to communicate proactively when issues happen, giving customers accurate information about the issue and the expected resolution time.

Keep your customers safe from cyber threats

The consequences of a data breach are deep and far-reaching, and they include financial losses, reputational damage, legal troubles, regulatory fines, and a loss of customer trust. Despite a greater emphasis on data security, hackers are constantly finding new ways to gain access to valuable corporate data and credentials by combining next-generation AI technologies with long-established tools.

Monitoring with Zabbix gives IT and security teams the visibility and early warning systems they need to spot and react to potential threats. Zabbix continuously monitors systems, networks, and applications for predefined thresholds and anomalies, identifying possible network intrusions or misconfigurations and notifying the relevant security stakeholders.

On top of that, Zabbix can monitor any existing security tools your team runs, tracking antivirus software, firewalls, IDS/IPS tools, and endpoint protection solutions to make sure they are functioning properly and running the latest versions. It can also integrate with SIEM systems (like Splunk, ELK, or Wazuh) as well as custom scripts in order to provide extended security analytics.

Meet (and exceed) your SLAs

Service Level Agreements (SLAs) are a framework for managing the expectations of both customers and businesses. They define agreed-on standards of service, but tracking them is more than just a way to measure compliance – it’s a tool that you can use to improve your overall service delivery and operations.

With Zabbix, you can monitor any quantifiable metric that’s relevant to your SLAs, such as system uptime/downtime, response time, the availability of web services, databases, or network devices, transaction success and failure rates, and much more. In addition, Zabbix can use real-time data and built-in SLA calculation to automatically calculate current SLA compliance and send an alert if an SLA is at risk of being breached, by using triggers based on thresholds.

If you’d rather track the metrics on your own, no problem – by using Zabbix dashboards, you can visualize SLA compliance in real-time, with the dashboards showing availability percentages, event timelines, and breach summaries, while giving you easy-to-understand views of service health. The result is better products and services that are aligned with customer expectations.

Build a continuous improvement culture

When it’s time to roll out a new feature or upgrade, you naturally want to have ALL the necessary data at your fingertips. Monitoring usage patterns and performance metrics with Zabbix not only gives you advanced visualizations (forecasting, capacity planning insights, etc.) but can also highlight cases where data analysis led to tangible improvements.

Want more input from customers and users? Zabbix can make sure that the improvements to your product are community-driven by giving you the data you need to run regular user surveys and forums to gather product feedback. It can even help you publish a public roadmap with transparent prioritization based on community input.

Conclusion

Customer satisfaction is about a lot more than just good service – it’s also about consistency, reliability, and transparency. Zabbix empowers businesses to deliver all three by providing a comprehensive, proactive, and scalable monitoring solution.

That’s why customers in verticals as diverse as aerospace and education turn to Zabbix to keep them informed about what’s working – and what isn’t. By integrating Zabbix into your IT operations, you’re not just improving system performance – you’re actively investing in customer satisfaction and loyalty.

Find out more about what Zabbix can do for you and your customers by taking a look at real-world case studies from companies like yours.

The post Improving Customer Satisfaction and Experience with Zabbix appeared first on Zabbix Blog.

Creating a Community-Driven Zabbix Book

2025-11-04 Zane Lasmane

Post Syndicated from Zane Lasmane original https://blog.zabbix.com/creating-a-community-driven-zabbix-book/31688/

At the recent Zabbix Summit community meeting, participants gathered to discuss an exciting initiative – the creation of the first-ever community-driven Zabbix book. While several books about Zabbix have been published in the past (often written by individual authors over a decade ago), this project marks a new milestone. For the first time, Zabbix community members from around the world are coming together to co-author a book, share their expertise, and tell the Zabbix story from many perspectives.

What is the Zabbix Book?

The project, hosted at thezabbixbook.com, is an open, collaborative effort led by Nathan Liefting and Patrik Uytterhoeven from Opensource ICT Solutions B.V. The goal is to create a community-built guide to Zabbix, written by users, for users. As Zabbix trainers, Patrik and Nathan have both been long-time (don’t want to say old) contributors to the Zabbix community, authoring multiple books and blog posts.

The Zabbix Book will cover topics ranging from cloud templates and infrastructure monitoring to host triggers, Zabbix internals, SNMP, low-level discovery, multi-factor authentication, and much more. Each contributor can choose a specific chapter or topic that matches their expertise, making it a truly collective and flexible effort.

The content is managed on GitHub, written in Markdown, and follows open contribution principles. The aim is to complete the main foundation of the book alongside the release of Zabbix 8.0 LTS (expected in 2026, Q1/Q2), with an update to include new 8.0 features approximately a month later.

Why write a Zabbix Book when documentation exists?

While the official Zabbix documentation remains the primary source for technical accuracy, the Zabbix Book serves as an alternative and more narrative approach to learning, created by everyday Zabbix users. It’s designed to introduce new users to Zabbix through practical examples, real-world use cases, and community wisdom – making it easier for newcomers to connect the dots.

How the community works together

During the Summit breakout session, the group discussed:

• The current project status and foundational setup
• How contributions are managed — commits, rules, and legal aspects
• Missing topics and a call for more writers, editors, and translators
• Ideas for practical information and real-world examples (like JMX, SNMP, etc.)
• Donations and funding goals, including ideas for supporting open-source projects, good causes, or new Zabbix community features

The project embraces an open, democratic spirit – anyone can contribute, vote, or help improve the book’s structure, content, and readability. The Zabbix Book is created by the Monitoring Penmasters Foundation, which was created in order to make it a real community project – all the intellectual rights belong to the foundation itself, and when revenue is created there will be a vote on where to donate the money.

Currently, the Monitoring Penmasters foundation consists of Patrik, Nathan, and Zabbix CEO and Founder Alexei Vladishev, who is involved in the book’s review and has agreed to contribute to some parts of the book while allocating design resources from Zabbix itself.

The project has also gotten a big assist from Brian van Baekel of Opensource ICT Solutions, a dedicated community member and certified Zabbix trainer who has given his fair share of presentations and written extensively about Zabbix and its capabilities.

Get involved

If you’d like to contribute, share your expertise, or simply follow the book’s progress, visit thezabbixbook.com to explore the current chapters and learn how to join the project. The project’s digital chapters are available to everyone, and while the writing and printing are still in progress, we hope to see finalized online and printed versions in spring 2026.

It’s also worth remembering that even though the book is free to download and use, the creators do have costs and financial contributions are welcome – you can chip in here.

Together, we’re not just writing a book — we’re writing a piece of Zabbix community history!

The post Creating a Community-Driven Zabbix Book appeared first on Zabbix Blog.

Monitoring a Starlink Dish with Zabbix

2025-10-21 Alexander Petrov-Gavrilov

Post Syndicated from Alexander Petrov-Gavrilov original https://blog.zabbix.com/monitoring-a-starlink-dish-with-zabbix/31543/

Did you realize that you can monitor a Starlink dish using just Zabbix? The idea (or rather the need) to use Starlink came to me almost as soon as I moved to a fairly rural area. Local internet providers have not yet “provided” fiberoptic or stable mobile connectivity to places like this, and while searching for a solution I accidentally discovered that Starlink was already providing service to some local companies. As I later found out, they also offered service in my area for residential customers.

To make a long story short, since internet access is crucial in the IT field, I decided to acquire and then monitor my very own Starlink dish. At first, this proved challenging because regular user data access is quite limited. However, thanks to Zabbix browser monitoring, I managed to solve it fairly easily. In this post I will share my solution with you, including the template.

Table of Contents

Monitoring configuration

First, you need to make sure you have Zabbix installed (either a Zabbix proxy or server) on the same network that the Starlink dish and router are on. The next step is to configure Zabbix for browser monitoring.

WebDriver installation

# podman run --name webdriver -d \
-p 4444:4444 \ 
-p 7900:7900 \
--shm-size="2g" \
--restart=always -d docker.io/selenium/standalone-chrome:latest

Port 4444 will be the port on which the WebDriver will be listening, and port 7900 will be used by NoVNC, which allows us to observe browser behavior in case a browser with a GUI is used.

Zabbix server/proxy configuration

After WebDriver is installed, we need to set up the communication between Zabbix and the driver. This can be done by editing the Zabbix server/proxy configuration file and updating the following parameters:

### Option: WebDriverURL 
# WebDriver interface HTTP[S] URL. For example http://localhost:4444 used with 
# Selenium WebDriver standalone server. 
# 
# WebDriverURL= 
WebDriverURL=http://localhost:4444 
### Option: StartBrowserPollers 
# Number of pre-forked instances of browser item pollers. 
# 
# Range: 0-1000 
# StartBrowserPollers=1 
StartBrowserPollers=5

With the configuration parameters in place, restart the Zabbix server/proxy to apply the changes:

systemctl restart zabbix-server

Creating a host

First, we need to navigate to the “Data collection” > “Hosts” section and create a host that represents our Starlink dish. The host in my example will look like this:

The host also has a user macro:

{$LINK} with value: http://webapp.starlink.com to point to the correct Starlink dish web app:

Creating a browser item

We will now configure our browser item to collect and monitor the list of metrics exposed in the Starlink browser app:

We are using the bare minimum here, so make sure the update intervals are as frequent as you need. However, I would not recommend updating it more frequently than every 5 minutes. It’s also not a good idea to store the history, since it is already stored trough dependent items.

The most important part of the item is the script itself:

var browser, result;
var opts = Browser.chromeOptions();

opts.capabilities.alwaysMatch['goog:chromeOptions'].args = [];
browser = new Browser(opts);
browser.setScreenSize(Number(1980), Number(1020));

try {
    var params = JSON.parse(value);
    browser.navigate(params.url);

 // Wait for the dish to report status
    Zabbix.sleep(2000);

    // Find the JSON text element(s)
    var jsonElements = browser.findElements("xpath", "//div[@id='root']/div[@class='App']/div[@class='Main']/div[2]/div[@class='Section'][2]/pre[@class='Json-Format']/div[@class='Json-Text']");
    var extractedData = [];

    for (var i = 0; i < jsonElements.length; i++) {
        var text = jsonElements[i].getText();

        // Try parsing JSON
        try {
            extractedData.push(JSON.parse(text));
        } catch (e) {
            // If not valid JSON, include raw text instead
            extractedData.push({ raw: text, error: "Invalid JSON format" });
        }
    }

    // Collect result 
    result = browser.getResult();

    // Replace with parsed JSON data
    result.extractedJsonData = extractedData.length === 1 ? extractedData[0] : extractedData;

}
catch (err) {
    if (!(err instanceof BrowserError)) {
        browser.setError(err.message);
    }
    result = browser.getResult();
}
finally {
    // Return a clean JSON object
    return JSON.stringify(result.extractedJsonData);
}

So what does this script do? It opens the Starlink web app, waits for the Starlink dish to output all the status data, and, after a bit of parsing, returns the data highlighted in the screenshot:

Now we can click on the three dots on the left of our newly created item in the items page and proceed to create dependent items for each value we are interested in!

Creating dependent items

Now we just click here:

As an example, to create an item that monitors the hardware version we can create an item like this:

With JSONPath preprocessing:

In the end we get the data in Zabbix:

All other items (except alerts) will follow the same logic – just update the item name, key, and JSONPath in preprocessing to extract the required values.

Creating dependent LLD item prototypes

To automate the alerts items creation, we can create a dependent discovery rule. In the “Discovery” section, create a new discovery rule:

With preprocessing using Java Script:

var data = JSON.parse(value);
var alerts = data.alerts;
var lld = [];

for (var key in alerts) {
    if (alerts.hasOwnProperty(key)) {
        lld.push({
            "{#ALERT}": key
        });
    }
}

return JSON.stringify({ data: lld });

This will provide us with following JSON data:

{
  "data": [
    {
      "{#ALERT}": "dishIsHeating"
    },
    {
      "{#ALERT}": "dishThermalThrottle"
    },
    {
      "{#ALERT}": "dishThermalShutdown"
    },
    {
      "{#ALERT}": "powerSupplyThermalThrottle"
    },
    {
      "{#ALERT}": "motorsStuck"
    },
    {
      "{#ALERT}": "mastNotNearVertical"
    },
    {
      "{#ALERT}": "slowEthernetSpeeds"
    },
    {
      "{#ALERT}": "softwareInstallPending"
    },
    {
      "{#ALERT}": "movingTooFastForPolicy"
    },
    {
      "{#ALERT}": "obstructed"
    }
  ]
}

All that’s left ‘to do is to create a dependent item prototype:

With preprocessing, of course:

JSONPath will transform to extract each specific alert and “Boolean to Decimal” will save us some space in the database by tranforming true/false booleans to digits.

Result

In the end, we can monitor all the data:

Even more data can be collected using exporters – if you are willing to do a bit of extra configuration, of course! Let me know if you are interested, and I will show you a completely different approach with a template.

Before I forget, the template used in this tutorial can be found here.

The post Monitoring a Starlink Dish with Zabbix appeared first on Zabbix Blog.

Community, Coffee, and Code: A Zabbix Summit 2025 Recap

2025-10-16 Michael Kammer

Post Syndicated from Michael Kammer original https://blog.zabbix.com/community-coffee-and-code-a-zabbix-summit-2025-recap/31577/

Zabbix Summit 2025 is officially in the history books, so now is the perfect time for a casual, behind‑the‑scenes run‑through of what went down. If you were there, this should ring a few bells (or spark some “oh hey, I forgot about that” moments). If you couldn’t make it, consider this your own personal highlight reel!

Featuring approximately 550 attendees from 42 countries, the Summit took place from October 8-10 at the Radisson Blu Hotel Latvija in the heart of downtown Riga. The 13th in-person version of our premier yearly event was in many ways our biggest and boldest yet, and it included keynote sessions, two parallel tracks (including a developer track), workshops, hands-on sessions, training and certification exams, and a variety of evening social and networking events.

Open source, open house

On October 8, we welcomed nearly 100 guests to our brand-new headquarters for Zabbix Summit 2025’s Open House Day. The new facility gave us plenty of space to host everyone, and visitors got to explore our new HQ, take part in a fun quiz with Zabbix facts, and catch up with longtime colleagues while meeting new ones from the community and the Zabbix team.

Day 1: Looking ahead

The Summit officially kicked off with Zabbix Founder and CEO Alexei Vladishev’s keynote address, entitled “Zabbix 8.0: A New Chapter in Monitoring.” The address laid out in detail what’s around the corner for Zabbix, including:

Zabbix Academy – a new learning hub with self-paced, expert-built courses to boost Zabbix skills anytime and from anywhere.
Zabbix France – Zabbix is acquiring IZI-IT and opening a new office in France to provide localized support and closer collaboration with French clients and partners.
Zabbix Cloud – a host of new features, including automatic upgrades and backups, plus predictable pricing and simplified user management.
Zabbix 8.0 LTS (coming in 2026) – a major leap forward with APM and OpenTelemetry for end-to-end visibility, Complex Event Processing (CEP) and AI-based correlation, plus new UI & visualizations for a smoother experience.
Zabbix Mobile App – coming with 8.0 LTS for iOS & Android, the app will offer instant push notifications, issue management, collaboration, seamless connection with Zabbix Cloud, and multi-server views in your pocket.
Zabbix Marketplace (2026) – A new global space to connect Zabbix users with vendor and partner solutions, Zabbix Marketplace will extend the power of Zabbix beyond our core product.

Next up was initMAX Founder and CEO Tomáš Heřmánek, who showed how to turn physical sensor data from analog inputs into Zabbix metrics with budget hardware and integrations, complete with templates and triggers.

Another crowd-pleasing session reached the audience thanks to Richard Germanus of CANCOM, who shared the story of how CANCOM consolidated six monitoring systems into one, managing approximately 30,000 hosts, deploying 162 Zabbix proxies, standardizing templates, integrating Power BI for dashboards, automating with APIs, and offering monitoring-as-a-service.

Shortly thereafter, a lightning talk by SEB Bank’s Giedrius Stasiulionis explored “Monitoring Sounds with Zabbix” – in other words, converting audio and sound waves into meaningful metrics, a fresh and inventive notion.

The day’s other lightning talk, “Monitor Your Nearby Areas and Events with Zabbix” by longtime Summit fixture and Zabbix superfan Janne Pikkarainen, showed how anyone can use Zabbix to centralize event data like train timetables, traffic patterns, or cinema showtimes.

Developer track: Something for everyone

Meanwhile, the Summit Developer track was full of special sessions for builders and extension authors, such as “Extend Zabbix Agent 2 with Your Plugin”, which saw Senior Golang Developer Eriks Sneiders show an appreciative audience how Zabbix agent 2’s plugin architecture works, how to use existing plugins, and how to build brand-new custom ones.

Other topics in the Developer track included template design, advanced scripting, API tips, and internal tooling, giving Zabbix techies some food for thought and hopefully sparking a batch of fresh ideas!

Day 2: Showing the big picture

After a long first day and night, Zabbix Summit 2025’s special guest Dylan Beattie made some noise and woke everyone up with a talk entitled “Open Source, Open Mind: The Cost of Free Software.”

Dylan took the Summit audience on a journey through the history and philosophy of free and open source software, touching on questions about licensing issues, looking at the motivations of developers, discussing edge cases and challenges, and asking whether truly sustainable open-source ecosystems can exist.

Later, Inqbeo Founder Christian Anton shared a system in which a central Zabbix instance serves multiple tenants, with the architecture leveraging Kafka to stream metric data partitioned per tenant, storing results in S3 (in Prometheus format), and visualizing via Grafana. This enables isolation and the creation of custom dashboards.

Other main-stage sessions tackled topics like scaling Zabbix, managing large datasets, tag and template strategies, and AI/automation in monitoring.

Connecting people with the Community track

Zabbix Summit 2025 also introduced a Community track, a dedicated space at Zabbix where users, enthusiasts, and contributors could share ideas and shape the future of Zabbix. Instead of deeply technical or development-level presentations, this track focused on community-driven topics like integrations, templates, connectors, media types, and open resources.

A key highlight was the “Zabbix Book Breakout Room”, led by Alexei Vladishev himself along with longtime community members Patrik Uytterhoeven, Brian van Baekel, and Nathan Liefting. Zabbix users were able to brainstorm ideas for new chapters, missing topics, translations, and community contributions to the online Zabbix Book.

Turning ideas into action

Day 2 was also full of hands-on workshops, including a fascinating one from the team at initMAX that was based on their day 1 presentation. Participants got kits with an ESP32 board, a camera, a 3D-printed counter mount, and a few other odds and ends. They were then guided step-by-step as they integrated the device into Zabbix, built monitoring scenarios, and used AI models to interpret camera images.

Meanwhile, the Summit also hosted training and certification exams before, during, and after the main event. Attendees could take courses like Automation & Integration with API, Database Monitoring, SNMP Monitoring, and level-up exams (Specialist and Professional) at discounted rates.

A different kind of networking

One of the things that makes the Zabbix Summit experience so special is the depth of the networking experience – there’s no awkward small talk or simple business card exchanges here, but rather a series of real connections made, deals closed, and new partnerships cemented.

Accordingly, a lot of the magic at Zabbix Summit 2025 happened after hours, with everyone gathering at Riga’s famed Monkey Club for the Summit Welcome Event on October 8 to enjoy a lively atmosphere, a wide selection of cocktails, and plenty of opportunities to connect with fellow monitoring and observability enthusiasts.

October 9’s Main Event took place in the Tallinn Quarter Angārs, which blended concert hall energy with an open-plan street food kitchen and bar that gave everyone plenty of room to mingle.

A special treat was provided in the form of an original Zabbix-related song by Zabbix PHP Developer and part-time rock star Vladimirs Maksimovs, which got the entire crowd on its feet and set the tone for an unforgettable evening.

In what has become a bit of a tradition within a tradition, the Summit officially wrapped up on October 10 at Riga’s Burzma Food Hall, with its relaxed atmosphere, multiple cuisines, and communal tables. It’s proven to be the perfect place for reflecting on Summit highlights, swapping contact info, or plotting collaborations.

Thank you to our sponsors!

We want to extend our heartfelt thanks to all the sponsors of Zabbix Summit 2025, whose commitment not only helped us bring everyone together under one roof but also contributed to the growth of both Zabbix and the entire global monitoring ecosystem. We value your partnership and look forward to working with you for many years to come!

initMAX – Diamond Sponsor
IntelliTrend – Platinum Sponsor
IZI-IT – Platinum Sponsor
Quadrata – Platinum Sponsor
rConfig – Platinum Sponsor
Allenta – Gold Sponsor
Metricio – Gold Sponsor
H5 Network – Gold Sponsor
Docomo Business – Silver Sponsor
SCSK – Silver Sponsor
Avantgarde Technologies – Lunch and coffee break sponsor

Thanks again to our sponsors and everyone else who helped make Zabbix Summit 2025 possible!

In case you couldn’t make it…

If you didn’t manage to make the trip, you can still enjoy the Summit atmosphere in the privacy of your own home! Recordings of both days are available on Zabbix’s YouTube channel:

Zabbix Summit 2025 Day 1 

Zabbix Summit 2025 Day 2 

The slides and texts of the presentations are also available here.

And that’s a wrap on Zabbix Summit 2025! From mind-blowing tech talks to caffeinated hallway chats and everything in between, this year’s Summit experience delivered. Whether you came for the deep dives or just the cool merch (no shame in that), we hope you went away inspired, connected, and maybe just a little more obsessed with monitoring and observability than before. See you in 2026!

The post Community, Coffee, and Code: A Zabbix Summit 2025 Recap appeared first on Zabbix Blog.

Introduction

What HyperPod resiliency already handles

Operational conditions where a human still wants to be in the loop

Opportunity: AWS DevOps Agent as a 24/7 companion

Solution Overview

What this solution delivers

Architecture

Event flow

​​Getting started​

Prerequisites

Deploying with CloudFormation

The webhook bridge — mapping HyperPod events to DevOps Agent

How the skills are defined — teaching the agent HyperPod’s operational model

The periodic-audit Lambda — Kubernetes state monitoring

Closing the loop — the email notifier

Viewing investigations in the DevOps Agent console

Asking the agent directly — the DevOps Agent Chat UI

Extending the solution — detection vs. reasoning

Investigation feedback

DevOps Agent APIs used by this solution

Cleaning up

Cost considerations

Conclusion

What’s next

About the authors

The sample workload

Prerequisites

Deploy the sample workload

Connect AWS DevOps Agent

How the alarm pipeline works

Run the scenarios

Scenario 1. Domain deny list blocking a legitimate endpoint

Scenario 2. Stateless rule priority misconfiguration

Scenario 3. Asymmetric cross Availability Zone routing drop

Further considerations

Clean up

Conclusion

The challenge

The solution

The results

Tune your database

Monitor the Zabbix database

Monitor the Zabbix server

Check the current state of your Zabbix server

Check the cache utilization

Consequences of running out of configuration cache

Consequences of running out of value and history cache

The result of cache tuning and database tuning

Tune the Zabbix server configuration parameters

Pitfalls of misconfiguration

Summary

Designed for real environments

Why it matters

How it looks

Open source

Onstage at Europol Industry and Research Days

Tops among 120 applicants

A first for Zabbix – and Latvia

Zabbix for law enforcement

Conclusion

The challenge

The solution

The results

Conclusion

What is an Agent Space and Why Does It Matter?

Part 1: Design your Agent Space architecture

Determine Your Agent Space Boundaries

Common Agent Space Patterns and Decision Points

Part 2: Implement your Agent Space architecture

Step 1: Agent Space Prerequisites

Step 2: Create an Agent Space

Step 3: Configure Integrations

Advanced Integrations

Step 4: Configure Access Controls

Step 5: Test and Iterate

Conclusion

Authors

Celebrating 20 years in business

A new path toward Zabbix expertise

Nous sommes Zabbix!

Getting started