Post Syndicated from Kaspars Mednis original https://blog.zabbix.com/decoding-zabbix-proxy-traffic-for-faster-troubleshooting/31898/

Usually, it is enough to simply look at the Zabbix proxy administration page or proxy health metrics to perform basic proxy troubleshooting. However, there are situations when a deeper look is required.

Today, we will examine the Zabbix server proxy communication and learn how to interpret the internal communication protocol.

Understanding the protocol

Zabbix communication protocol

Zabbix components use TCP for communication, and information is encoded in JSON. How do you distinguish Zabbix communication packets? There are a few main filters you need to apply:

• Protocol: TCP

• Port: 10051 or 10050 (depending on whether components are active or passive)

• Packet: Starts with ZBXD or 5A 42 58 44 in HEX

On older versions, it was simple to capture and read Zabbix packets in plain text. Starting with Zabbix 4.0.0, mandatory traffic compression was implemented. This greatly reduces network traffic – roughly by 10× with negligible CPU overhead, but it also makes the traffic unreadable to humans.

A modern Zabbix communication packet looks like this:

5a425844038200000097000000789c2dcccb0e83201085e15731b33606b90a8fe20ec631256da4056a6c9abe7be965fb7f27e709996e772a151c5c733a1edde2ab871e4ee9db661f423cba1f79ac71a786854a89696bbe7223e5b2326b75e01c199368510b42cf68f2eaf3b453fe8fcdc0063eb68497846770a3d1c25a698cea612be0b43642a9c9b2d71b6c5d2cfd

Not very human-friendly, right? In the following sections we will capture and decompress this communication packet step by step.

Capturing traffic

There are multiple tools available for this purpose, but we will use Wireshark – one of the most popular and widely used packet analysis tools. It provides a nice graphical interface for Windows and Linux, but we will use the command-line version, since most troubleshooting is performed over an SSH session. The system used in this example is CentOS Stream 9, but the commands should work on other Linux distributions with only minor syntax adjustments.

First, install the tool:

dnf install wireshark-cli

This installs the tshark command-line utility. After that, change your working directory to a location where you can write files. In this example, we will use /tmp:

cd /tmp

Next, let’s capture some traffic between the Zabbix server and an active proxy:

tshark -i eth0 -f "host <ZABBIX SERVER IP> and host <ZABBIX PROXY IP> \
and tcp port 10051" -w zabbix_stream.pcap

Explanation of parameters:

• -i eth0 – listen on interface eth0 (specify a different interface if needed)

• <ZABBIX SERVER IP> – replace with the Zabbix server IP address

• <ZABBIX PROXY IP> – replace with the Zabbix proxy IP address

• tcp port 10051 – capture TCP packets on port 10051 (Zabbix trapper)

• -w zabbix_stream.pcap – write captured output to a file

Let this run for a couple of minutes to collect some raw traffic data. Press CTRL + C to stop the capture.

Analyzing capture file

Now we have captured a *.pcap file that contains multiple TCP streams. A TCP stream represents a single TCP connection. Since Zabbix proxies do not keep persistent connections and instead open a new connection whenever needed, a Zabbix active proxy typically produces the following streams:

• Data sender – sends collected values every second (by default)

• Configuration syncer – downloads configuration updates every 10 seconds (by default)

To view the contents of the *.pcap file, run:

tshark -r zabbix_stream.pcap -q -z conv,tcp

Example output:

TCP Conversations
Filter:<No Filter>
                                   |      <-    ||      ->    ||     Total   |Relative|
                                   |Frames Bytes||Frames Bytes||Frames Bytes |Start   |       
10.10.0.2:57850 <-> 10.20.0.5:10051 5 2,512bytes  6 547bytes    11 3,059bytes 0.0000   
10.10.0.2:57860 <-> 10.20.0.5:10051 5 399bytes    5 516bytes    10 915bytes   0.4700  
10.10.0.2:57864 <-> 10.20.0.5:10051 5 399bytes    5 521bytes    10 920bytes   1.4768  
10.10.0.2:57876 <-> 10.20.0.5:10051 5 399bytes    5 570bytes    10 969bytes   2.4829   
10.10.0.2:57878 <-> 10.20.0.5:10051 5 399bytes    5 522bytes    10 921bytes   3.4882   
10.10.0.2:46628 <-> 10.20.0.5:10051 5 399bytes    5 527bytes    10 926bytes   4.4935   
10.10.0.2:46642 <-> 10.20.0.5:10051 4 333bytes    6 590bytes    10 923bytes   5.4992   
10.10.0.2:46648 <-> 10.20.0.5:10051 5 399bytes    5 478bytes    10 877bytes   6.5047   
10.10.0.2:46662 <-> 10.20.0.5:10051 5 399bytes    5 480bytes    10 879bytes   7.5097

tshark -r zabbix_stream.pcap -T fields \
-e tcp.stream -e frame.number -e frame.time_relative -e frame.len

0 1  0.000000000 76
0 2  0.000005109 76
0 3  0.000078403 68
0 4  0.000079579 68
0 5  0.000280946 209
0 6  0.000283835 209
0 7  0.001188322 68
0 8  0.001189912 68
0 9  0.001421210 68
0 10 0.001422856 68
1 11 1.003582601 76
1 12 1.003588266 76
1 13 1.003646494 68
1 14 1.003647585 68
1 15 1.003741654 256
1 16 1.003758183 256
1 17 1.004531106 68
1 18 1.004532827 68
1 19 1.004973531 68
.....

tshark -r zabbix_stream.pcap -T fields \
-e tcp.stream -e frame.number -e frame.time_relative -e frame.len -e tcp.payload

0 1  0.000000000 76
0 2  0.000005109 76
0 3  0.000078403 68
0 4  0.000079579 68
0 5  0.000280946 209 5a425844038000000096000000789c2dca4d0e82301040e1ab90591352fb3703477137d3d6483454692518e3dd6dd4edfbde0bd6747fa4526182db9af76717b932f470cedf76649179ef7ec4a1ce5b6a585229735e9a2bf12aa2481c89299032c2ceb21754a44fda609bb7b4fe671cece05a09d71c2e301dd05b4548daf4b014984667b52734f6fd013eac2c96
0 6  0.000283835 209 5a425844038000000096000000789c2dca4d0e82301040e1ab90591352fb3703477137d3d6483454692518e3dd6dd4edfbde0bd6747fa4526182db9af76717b932f470cedf76649179ef7ec4a1ce5b6a585229735e9a2bf12aa2481c89299032c2ceb21754a44fda609bb7b4fe671cece05a09d71c2e301dd05b4548daf4b014984667b52734f6fd013eac2c96
0 7  0.001188322 68
0 8  0.001189912 68
0 9  0.001421210 68
0 10 0.001422856 68
......

5a42584403af000000f0000000789c658ecb0e823014447f85dc3521853e6edb4fd1b868a1c646b44a0bc110fedd22ec5cce9ce4cc2c30b8f7e862020daf21cc9fa233c94009b7f0eb4ec65a3f173b326df293cb30ba187d78664eac201d5adb2969642b09b58633232c12d95c1b8a9bc9c7148643accf0bf80e34150d2bc127f7d812278cd312da3eb477d0350a4624ca2657cf081a15e74cd588254ca61f5d9ead61bde4e486e30656ace2f06f60bb417154825056af5fed34456b

tshark -r zabbix_stream.pcap -T fields -e tcp.payload -E occurrence=f \
| grep -v '^$'

5a425844038000000096000000789c2dca4d0e82301040e1ab90591352fb3703477137d3d6483454692518e3dd6dd4edfbde0bd6747fa4526182db9af76717b932f470cedf76649179ef7ec4a1ce5b6a585229735e9a2bf12aa2481c89299032c2ceb21754a44fda609bb7b4fe671cece05a09d71c2e301dd05b4548daf4b014984667b52734f6fd013eac2c96                                                                                5a425844038000000096000000789c2dca4d0e82301040e1ab90591352fb3703477137d3d6483454692518e3dd6dd4edfbde0bd6747fa4526182db9af76717b932f470cedf76649179ef7ec4a1ce5b6a585229735e9a2bf12aa2481c89299032c2ceb21754a44fda609bb7b4fe671cece05a09d71c2e301dd05b4548daf4b014984667b52734f6fd013eac2c96                                                                                5a42584403af000000f0000000789c658ecb0e823014447f85dc3521853e6edb4fd1b868a1c646b44a0bc110fedd22ec5cce9ce4cc2c30b8f7e862020daf21cc9fa233c94009b7f0eb4ec65a3f173b326df293cb30ba187d78664eac201d5adb2969642b09b58633232c12d95c1b8a9bc9c7148643a

tshark -r zabbix_stream.pcap -T fields -e tcp.payload -E occurrence=f \
| grep -v '^$'  > zabbix_payload.hex

#!/usr/bin/python3
import zlib

hex_file = "zabbix_payload.hex"
ZBXD_HEADER_LEN = 26 # 13 bytes * 2 hex chars per byte

with open(hex_file, "r") as f:
  for line_number, line in enumerate(f, 1):
    line = line.strip()
    if not line:
      continue

    # Remove Zabbix header
    if line.startswith("5a425844"):
      payload_hex = line[ZBXD_HEADER_LEN:]
    else:
      payload_hex = line

    # Convert hex to bytes
    try:
      payload_bytes = bytes.fromhex(payload_hex)
    except ValueError as e:
      print(f"Line {line_number}: Invalid hex, skipping ({e})")
      continue

    # Decompress using zlib
    try:
      decompressed = zlib.decompress(payload_bytes)
    except zlib.error as e:
      print(f"Line {line_number}: Decompression error ({e})")
      continue
  
    print(f"Line {line_number}: {decompressed}")

chmod +x decompress.py

./decompress.py

Line 59: b'{"request":"proxy data","host":"Zabbix proxy active","session":"fbdb545d8250bb4c9b2341cc8ca055f1","history data":[{"id":13,"itemid":50454,"clock":1764172374,"ns":946257883,"value":"[{\\"{#IFNAME}\\":\\"lo\\"},{\\"{#IFNAME}\\":\\"eth0\\"}]"}],"version":"7.4.5","clock":1764172375,"ns":432069960}'
Line 60: b'{"upload":"enabled","response":"success","tasks":[{"type":6,"clock":1764172373,"ttl":3600,"itemid":50454}]}'
Line 61: b'{"request":"proxy data","host":"Zabbix proxy active","session":"fbdb545d8250bb4c9b2341cc8ca055f1","version":"7.4.5","clock":1764172375,"ns":438122213}'
Line 62: b'{"upload":"enabled","response":"success"}'
Line 63: b'{"request":"proxy config","host":"Zabbix proxy active","version":"7.4.5","session":"fbdb545d8250bb4c9b2341cc8ca055f1", "config_revision":18611,"proxy_secrets_provider":0}'
Line 64: b'{"data":{},"config_revision":18613}'

tshark -i eth0 -f "host <ZABBIX SERVER IP> and host <ZABBIX PROXY IP> \
and tcp port 10051" -w zabbix_stream.pcap

tshark -r zabbix_stream.pcap -T fields -e tcp.payload -E occurrence=f \
| grep -v '^$' > zabbix_payload.hex

./decompress.py

tshark -i eth0 -f "host <ZABBIX SERVER IP> and host <ZABBIX PROXY IP> \
and tcp port 10051" -w zabbix_stream.pcap

tshark -r zabbix_stream.pcap -T fields -e tcp.stream -e tcp.payload \
-E occurrence=f -E separator=, -E quote=d, -Y 'tcp.payload && tcp.payload != ""' \
> zabbix_payload.csv

"2","5a42584403aa000000dd000000789c458d410e83201444af62fe9a1814a896a3b4e9e283df9494480bd4688c772f694dba9d37336f8348af37a50c1a9e312c6b35604660700fdfec82c6b8a5fa21b4d9cd5460a2945c980a1fcd60945443df2a6e8cb467d30ad958db5be44a8d4d29bb29531cd15285333a8fc6799757d0d7ed8fdc005a080647c31368ce80620cb14860bf3198291eceae96b52ac7d607fb00dd7427d974ad506531a5f2c3c599ab9ecbfd038a0944ee" "2","5a425844033000000029000000789cab562a2dc8c94f4c51b2524acd4b4cca494d51d2512a4a2d2ec8cf2b4e050a16972627a716172bd502002b010e61" "3","5a42584403db0000003d010000789c658fdd6ac3300c855f25e8da143bb6f2e317196cecc23f0a33f3e2cd76434be9bbcf4d03bbd89584bea3a3a31b64fa3953a9a0e13ba7cbb5f3a61a60f091f6d9abb1365cba2732ae868d1a2c544a486be38bf51615faa9476ead72b3eda512ce4dce70c4453471582be5c538eacc66423436c450afa0df6e7f2878d05232381491400b069473caed08dcdf5ba0506aca47be7dd9efa250e9ebd12257c819b898dc6703e3a0c4d8cbc7682da06735e1340e02196c269e9b3fbc90ed0ae58df2eedfeaf1d378522784ff56e2692545afe6990fc3fd17684060c8" "3","5a425844033000000029000000789cab562a2dc8c94f4c51b2524acd4b4cca494d51d2512a4a2d2ec8cf2b4e050a16972627a716172bd502002b010e61"

Next, let’s create a slightly modified Python script to display the entries per stream. Name it streams.py:

#!/usr/bin/python3

import csv
import zlib
import json

csv_file = "zabbix_payload.csv"
ZBXD_HEADER_LEN = 26 # 13 bytes * 2 hex chars per byte
streams = {}
with open(csv_file, "r") as f:
  reader = csv.reader(f)
  for row_number, row in enumerate(reader, 1):
    if len(row) < 2:
       continue

    stream_id = row[0].strip().strip('"')
    hexdata = row[1].strip().strip('"')

    if not hexdata:
      continue

    # Remove Zabbix header
    if hexdata.startswith("5a425844"):
      hex_payload = hexdata[ZBXD_HEADER_LEN:]
    else:
      hex_payload = hexdata

    # Convert hex to bytes
    try:
      payload_bytes = bytes.fromhex(hex_payload)
    except ValueError as e:
      print(f"[Line {row_number}] Invalid hex: {e}")
      continue

    # Decompress
    try:
      decompressed = zlib.decompress(payload_bytes)
    except zlib.error as e:
      print(f"[Line {row_number}] Decompression error: {e}")
      continue

    # Store in the stream bucket
    streams.setdefault(stream_id, []).append(decompressed)

# ---- OUTPUT SECTION ----

print("\n===== STREAM PAIRS =====\n")

for stream_id, messages in streams.items():
  print(f"=== Stream {stream_id} ===")
  for i, msg in enumerate(messages):
    label = (
      "Request:" if i == 0
      else "Response:" if i == 1
      else f"Extra message #{i+1}:"
    )
    print(label)
    text = msg.decode("utf-8")

    # Try to pretty-print JSON
    try:
      parsed = json.loads(text)
      pretty_json = json.dumps(parsed, indent=4, ensure_ascii=False)
      print(pretty_json)
    except json.JSONDecodeError:
    # fallback: print raw text
      print(text)
    print()

Make the file executable:

chmod +x streams.py

Execute the file:

./streams.py

The script will output decompressed Zabbix traffic in a parsed JSON format:

=== Stream 0 ===
Request:
{
  "request": "proxy data",
  "host": "Zabbix proxy active",
  "session": "fbdb545d8250bb4c9b2341cc8ca055f1",
  "interface availability": [
    {
      "interfaceid": 33,
      "available": 0,
      "error": ""
    }
  ],
  "version": "7.4.5",
  "clock": 1764172350,
  "ns": 303905804
}
Response:
{
  "upload": "enabled",
  "response": "success"
}

=== Stream 1 ===
Request:
.......

You’ll notice that typical communication produces two entries per stream – one request from the Zabbix proxy and one response from the Zabbix server. With this approach, it’s much easier to understand and troubleshoot the communication – all traffic is now grouped into request-response pairs and presented in a clean, formatted way.

Live data

And finally — can we make all of this run live? Absolutely, with a little help from our third Python script. The previous two examples walked through the workflow step by step: capture → extract payload → decompress. Now everything comes together in a single script that handles the entire process for you.

Create a new file named live.py:

#!/usr/bin/python3

import subprocess
import zlib
import json
from datetime import datetime

ZBXD_HEADER_LEN = 26 # 13 bytes * 2 hex chars

# === Configurable parameters ===
SRC_IP = "161.35.217.186"
DST_IP = "134.209.233.72"
TCP_PORT = "10051"
INTERFACE = "eth0"

tshark_cmd = [
  "tshark",
  "-i", INTERFACE,
  "-l",
  "-f", f"host {SRC_IP} and host {DST_IP} and tcp port {TCP_PORT}",
  "-T", "fields",
  "-e", "tcp.stream",
  "-e", "tcp.payload",
  "-E", "separator=,",
  "-E", "quote=d",
  "-E", "occurrence=f",
  "-Y", "tcp.payload && tcp.payload != \"\""
]

proc = subprocess.Popen(
  tshark_cmd,
  stdout=subprocess.PIPE,
  stderr=subprocess.DEVNULL,
  text=True
)

seen_streams = set() # track streams we've already printed

for line in proc.stdout:
  line = line.strip()
  if not line:
    continue

  # Split CSV (stream_number, payload_hex)
  try:
    stream_num, payload_hex = line.split(",", 1)
    payload_hex = payload_hex.strip('"')
  except ValueError:
    continue

  # Only print timestamp once per stream
  if stream_num not in seen_streams:
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]
    print(f"\n=== [{timestamp}] Stream {stream_num} ===")
    seen_streams.add(stream_num)

  # Remove Zabbix header
  if payload_hex.startswith("5a425844"):
    payload_hex = payload_hex[ZBXD_HEADER_LEN:]

  # Convert hex to bytes
  try:
    payload_bytes = bytes.fromhex(payload_hex)
  except ValueError:
    continue

  # Decompress
  try:
    decompressed = zlib.decompress(payload_bytes)
  except zlib.error:
    continue

  # Pretty print JSON if possible
  try:
    json_obj = json.loads(decompressed)
    pretty = json.dumps(json_obj, indent=2)
    print(pretty)
  except json.JSONDecodeError:
    print(decompressed)

Make the file executable:

chmod +x live.py

Execute the file:

./live.py

And that’s it – your script now watches live proxy traffic and streams the output as JSON. Pretty cool, right?

=== [2025-11-27 16:59:31.593] Stream "0" ===
{
  "request": "proxy data",
  "host": "Zabbix proxy active",
  "session": "fbdb545d8250bb4c9b2341cc8ca055f1",
  "history data": [
    {
      "id": 73726,
      "itemid": 50459,
      "clock": 1764262769,
      "ns": 947018320,
      "value": "0"
    },
    {
      "id": 73727,
      "itemid": 50450,
      "clock": 1764262770,
      "ns": 947145177
    }
  ],
  "version": "7.4.5",
  "clock": 1764262770,
  "ns": 961735298
}
{
  "upload": "enabled",
"  response": "success"
}
.....

Final notes

The example scripts provided here are for demonstration purposes only, tested in a small demo environment. While the same principles apply to larger setups, keep in mind that proxies in production can handle hundreds or even thousands of new values per second (NVPS), which significantly increases the payload volume. Also, all examples assume a Zabbix proxy running in active mode – passive proxies communicate slightly differently. A similar approach can be used to monitor Zabbix Agent communications.

So, what valuable information can you actually gather from Zabbix proxy ⇄ Zabbix Server communication?

• The types of data sent from proxy to server

• Configuration updates and their contents

• Test and Execute Now tasks

• Discovery and Autoregistration data

If you’re interested in exploring discovery, autoregistration, encryption, or other aspects of Zabbix’s internal communication, feel free to leave a comment!

The post Decoding Zabbix Proxy Traffic for Faster Troubleshooting appeared first on Zabbix Blog.