Post Syndicated from Kaspars Mednis original https://blog.zabbix.com/decoding-zabbix-proxy-traffic-for-faster-troubleshooting/31898/
Usually, it is enough to simply look at the Zabbix proxy administration page or proxy health metrics to perform basic proxy troubleshooting. However, there are situations when a deeper look is required.
Today, we will examine the Zabbix server 
proxy communication and learn how to interpret the internal communication protocol.
Understanding the protocol
Zabbix communication protocol
Zabbix components use TCP for communication, and information is encoded in JSON. How do you distinguish Zabbix communication packets? There are a few main filters you need to apply:
-
Protocol: TCP
-
Port: 10051 or 10050 (depending on whether components are active or passive)
-
Packet: Starts with
ZBXDor5A 42 58 44in HEX
On older versions, it was simple to capture and read Zabbix packets in plain text. Starting with Zabbix 4.0.0, mandatory traffic compression was implemented. This greatly reduces network traffic – roughly by 10× with negligible CPU overhead, but it also makes the traffic unreadable to humans.
A modern Zabbix communication packet looks like this:
5a425844038200000097000000789c2dcccb0e83201085e15731b33606b90a8fe20ec631256da4056a6c9abe7be965fb7f27e709996e772a151c5c733a1edde2ab871e4ee9db661f423cba1f79ac71a786854a89696bbe7223e5b2326b75e01c199368510b42cf68f2eaf3b453fe8fcdc0063eb68497846770a3d1c25a698cea612be0b43642a9c9b2d71b6c5d2cfd
Not very human-friendly, right? In the following sections we will capture and decompress this communication packet step by step.
Capturing traffic
There are multiple tools available for this purpose, but we will use Wireshark – one of the most popular and widely used packet analysis tools. It provides a nice graphical interface for Windows and Linux, but we will use the command-line version, since most troubleshooting is performed over an SSH session. The system used in this example is CentOS Stream 9, but the commands should work on other Linux distributions with only minor syntax adjustments.
First, install the tool:
dnf install wireshark-cli
This installs the tshark command-line utility. After that, change your working directory to a location where you can write files. In this example, we will use /tmp:
cd /tmp
Next, let’s capture some traffic between the Zabbix server and an active proxy:
tshark -i eth0 -f "host <ZABBIX SERVER IP> and host <ZABBIX PROXY IP> \ and tcp port 10051" -w zabbix_stream.pcap
Explanation of parameters:
-
-i eth0– listen on interface eth0 (specify a different interface if needed) -
<ZABBIX SERVER IP>– replace with the Zabbix server IP address -
<ZABBIX PROXY IP>– replace with the Zabbix proxy IP address -
tcp port 10051– capture TCP packets on port 10051 (Zabbix trapper) -
-w zabbix_stream.pcap– write captured output to a file
Let this run for a couple of minutes to collect some raw traffic data. Press CTRL + C to stop the capture.
Analyzing capture file
Now we have captured a *.pcap file that contains multiple TCP streams. A TCP stream represents a single TCP connection. Since Zabbix proxies do not keep persistent connections and instead open a new connection whenever needed, a Zabbix active proxy typically produces the following streams:
-
Data sender – sends collected values every second (by default)
-
Configuration syncer – downloads configuration updates every 10 seconds (by default)
To view the contents of the *.pcap file, run:
tshark -r zabbix_stream.pcap -q -z conv,tcp
Example output:
tshark -r zabbix_stream.pcap -T fields \ -e tcp.stream -e frame.number -e frame.time_relative -e frame.len
"2","5a42584403aa000000dd000000789c458d410e83201444af62fe9a1814a896a3b4e9e283df9494480bd4688c772f694dba9d37336f8348af37a50c1a9e312c6b35604660700fdfec82c6b8a5fa21b4d9cd5460a2945c980a1fcd60945443df2a6e8cb467d30ad958db5be44a8d4d29bb29531cd15285333a8fc6799757d0d7ed8fdc005a080647c31368ce80620cb14860bf3198291eceae96b52ac7d607fb00dd7427d974ad506531a5f2c3c599ab9ecbfd038a0944ee" "2","5a425844033000000029000000789cab562a2dc8c94f4c51b2524acd4b4cca494d51d2512a4a2d2ec8cf2b4e050a16972627a716172bd502002b010e61" "3","5a42584403db0000003d010000789c658fdd6ac3300c855f25e8da143bb6f2e317196cecc23f0a33f3e2cd76434be9bbcf4d03bbd89584bea3a3a31b64fa3953a9a0e13ba7cbb5f3a61a60f091f6d9abb1365cba2732ae868d1a2c544a486be38bf51615faa9476ead72b3eda512ce4dce70c4453471582be5c538eacc66423436c450afa0df6e7f2878d05232381491400b069473caed08dcdf5ba0506aca47be7dd9efa250e9ebd12257c819b898dc6703e3a0c4d8cbc7682da06735e1340e02196c269e9b3fbc90ed0ae58df2eedfeaf1d378522784ff56e2692545afe6990fc3fd17684060c8" "3","5a425844033000000029000000789cab562a2dc8c94f4c51b2524acd4b4cca494d51d2512a4a2d2ec8cf2b4e050a16972627a716172bd502002b010e61"
Next, let’s create a slightly modified Python script to display the entries per stream. Name it streams.py:
#!/usr/bin/python3
import csv
import zlib
import json
csv_file = "zabbix_payload.csv"
ZBXD_HEADER_LEN = 26 # 13 bytes * 2 hex chars per byte
streams = {}
with open(csv_file, "r") as f:
reader = csv.reader(f)
for row_number, row in enumerate(reader, 1):
if len(row) < 2:
continue
stream_id = row[0].strip().strip('"')
hexdata = row[1].strip().strip('"')
if not hexdata:
continue
# Remove Zabbix header
if hexdata.startswith("5a425844"):
hex_payload = hexdata[ZBXD_HEADER_LEN:]
else:
hex_payload = hexdata
# Convert hex to bytes
try:
payload_bytes = bytes.fromhex(hex_payload)
except ValueError as e:
print(f"[Line {row_number}] Invalid hex: {e}")
continue
# Decompress
try:
decompressed = zlib.decompress(payload_bytes)
except zlib.error as e:
print(f"[Line {row_number}] Decompression error: {e}")
continue
# Store in the stream bucket
streams.setdefault(stream_id, []).append(decompressed)
# ---- OUTPUT SECTION ----
print("\n===== STREAM PAIRS =====\n")
for stream_id, messages in streams.items():
print(f"=== Stream {stream_id} ===")
for i, msg in enumerate(messages):
label = (
"Request:" if i == 0
else "Response:" if i == 1
else f"Extra message #{i+1}:"
)
print(label)
text = msg.decode("utf-8")
# Try to pretty-print JSON
try:
parsed = json.loads(text)
pretty_json = json.dumps(parsed, indent=4, ensure_ascii=False)
print(pretty_json)
except json.JSONDecodeError:
# fallback: print raw text
print(text)
print()
Make the file executable:
chmod +x streams.py
Execute the file:
./streams.py
The script will output decompressed Zabbix traffic in a parsed JSON format:
=== Stream 0 ===
Request:
{
"request": "proxy data",
"host": "Zabbix proxy active",
"session": "fbdb545d8250bb4c9b2341cc8ca055f1",
"interface availability": [
{
"interfaceid": 33,
"available": 0,
"error": ""
}
],
"version": "7.4.5",
"clock": 1764172350,
"ns": 303905804
}
Response:
{
"upload": "enabled",
"response": "success"
}
=== Stream 1 ===
Request:
.......
You’ll notice that typical communication produces two entries per stream – one request from the Zabbix proxy and one response from the Zabbix server. With this approach, it’s much easier to understand and troubleshoot the communication – all traffic is now grouped into request-response pairs and presented in a clean, formatted way.
Live data
And finally — can we make all of this run live? Absolutely, with a little help from our third Python script. The previous two examples walked through the workflow step by step: capture → extract payload → decompress. Now everything comes together in a single script that handles the entire process for you.
Create a new file named live.py:
#!/usr/bin/python3
import subprocess
import zlib
import json
from datetime import datetime
ZBXD_HEADER_LEN = 26 # 13 bytes * 2 hex chars
# === Configurable parameters ===
SRC_IP = "161.35.217.186"
DST_IP = "134.209.233.72"
TCP_PORT = "10051"
INTERFACE = "eth0"
tshark_cmd = [
"tshark",
"-i", INTERFACE,
"-l",
"-f", f"host {SRC_IP} and host {DST_IP} and tcp port {TCP_PORT}",
"-T", "fields",
"-e", "tcp.stream",
"-e", "tcp.payload",
"-E", "separator=,",
"-E", "quote=d",
"-E", "occurrence=f",
"-Y", "tcp.payload && tcp.payload != \"\""
]
proc = subprocess.Popen(
tshark_cmd,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
text=True
)
seen_streams = set() # track streams we've already printed
for line in proc.stdout:
line = line.strip()
if not line:
continue
# Split CSV (stream_number, payload_hex)
try:
stream_num, payload_hex = line.split(",", 1)
payload_hex = payload_hex.strip('"')
except ValueError:
continue
# Only print timestamp once per stream
if stream_num not in seen_streams:
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]
print(f"\n=== [{timestamp}] Stream {stream_num} ===")
seen_streams.add(stream_num)
# Remove Zabbix header
if payload_hex.startswith("5a425844"):
payload_hex = payload_hex[ZBXD_HEADER_LEN:]
# Convert hex to bytes
try:
payload_bytes = bytes.fromhex(payload_hex)
except ValueError:
continue
# Decompress
try:
decompressed = zlib.decompress(payload_bytes)
except zlib.error:
continue
# Pretty print JSON if possible
try:
json_obj = json.loads(decompressed)
pretty = json.dumps(json_obj, indent=2)
print(pretty)
except json.JSONDecodeError:
print(decompressed)
Make the file executable:
chmod +x live.py
Execute the file:
./live.py
And that’s it – your script now watches live proxy traffic and streams the output as JSON. Pretty cool, right?
=== [2025-11-27 16:59:31.593] Stream "0" ===
{
"request": "proxy data",
"host": "Zabbix proxy active",
"session": "fbdb545d8250bb4c9b2341cc8ca055f1",
"history data": [
{
"id": 73726,
"itemid": 50459,
"clock": 1764262769,
"ns": 947018320,
"value": "0"
},
{
"id": 73727,
"itemid": 50450,
"clock": 1764262770,
"ns": 947145177
}
],
"version": "7.4.5",
"clock": 1764262770,
"ns": 961735298
}
{
"upload": "enabled",
" response": "success"
}
.....
Final notes
The example scripts provided here are for demonstration purposes only, tested in a small demo environment. While the same principles apply to larger setups, keep in mind that proxies in production can handle hundreds or even thousands of new values per second (NVPS), which significantly increases the payload volume. Also, all examples assume a Zabbix proxy running in active mode – passive proxies communicate slightly differently. A similar approach can be used to monitor Zabbix Agent communications.
So, what valuable information can you actually gather from Zabbix proxy ⇄ Zabbix Server communication?
-
The types of data sent from proxy to server
-
Configuration updates and their contents
-
Test and Execute Now tasks
-
Discovery and Autoregistration data
If you’re interested in exploring discovery, autoregistration, encryption, or other aspects of Zabbix’s internal communication, feel free to leave a comment!
The post Decoding Zabbix Proxy Traffic for Faster Troubleshooting appeared first on Zabbix Blog.
