As Zabbix 7.0 will come with the new widget framework, allowing communication between different widgets on dashboards, of course, I had to try it out.
Creating the module
The blog post title is a bit of a clickbait in the sense that this example is just 1:1 from the Zabbix Summit 2023 custom widgets workshop session. I made some very, very minor modifications to the code, mainly just changing my name and so on to manifest.json files. Since the code itself was obtained from the workshop session, I’m not going to publish it, but this much I will tease:
Beginning with Zabbix 7.0, you can create your own custom widgets with JavaScript & PHP, and easily make other widgets on the dashboard to react to clicks you made on some other widget. The manifest.json file in the root of your custom module can describe what kind of info your widget will broadcast to other dashboard widgets, or what kind of info it will be receiving and obeying. Other than that, the custom widget only has a 2.6-kilobyte JavaScript file and a 587-byte CSS file. Modules are placed under /usr/share/zabbix/modules.
Next, just like in older versions of Zabbix, to activate your module you just go to Administration->Modules and click on Scan modules. And, there you have it.
Then, in your widgets, you can enable the dynamic reactions to other widgets or dashboard query changes like this:
Great! But what will it do?
I now have a new way of filtering the visible alerts. The custom widget on the left lists my host groups and hosts that belong to them.
Observe what happens when I click on the Electricity usage button:
I’m not limited to only selecting one host at a time, I can click on multiple hosts. Now see what happens if I also choose Lunch menus from my hosts.
The possibilities are endless
This example is just a simple read-only example. But, as Alexei mentioned to me after my speech at the Zabbix Summit 2023, this new framework could be used for controlling stuff, too. When I have time, I’ll try to run custom scripts and do other write operations through Zabbix API and this new framework.
Having a proper control panel for switching on/off the lights, music, and other things would be really cool, and now it certainly is possible. The future of the Zabbix user interface is really exciting thanks to new custom widgets.
This post was originally published on the author’s page.
Since I will have some real use for Zabbix 7.0 when it comes out, I figured out that maybe it’s time to switch my What’s up, home? main instance to run on Zabbix 7.0beta1.
Actually, I first upgraded to Zabbix 7.0alpha9 early yesterday, but then 7.0beta1 got released later in the evening before I had time to play around with alpha9.
Anyway, now my Raspberry Pi 4 is running the latest and greatest version of Zabbix. A possible bumpy ride ahead, but I’m ready!
First impressions
The upgrade process itself went smoothly, just like with the stable releases. All my data, dashboards, triggers, and other rules are still in place.
Developers tell you that the new 7.0 will be much faster under the hood due to migrating to threads and asynchronous polling, among other changes. It ain’t just market speak, as this is my Zabbix instance before and after the upgrade. I don’t think that I need to annotate the graphs to show the point when I did the upgrade. The part that’s still hovering around 20% is my ICMP ping pollers. Other than that, in my humble home setup, everything is now pretty much idle.
Looking at my Raspberry Pi dashboard, not much has changed, and anyway, my Raspberry Pi is running many other things than Zabbix, too.
Here’s CPU:
Memory:
Disk I/O utilization:
Temperature:
From single item view to gauges
To try out the new gauge widget, I threw in a few of them showing some temperatures. The widget is very configurable.
Interactive manual host/event actions
In addition to being actually useful in production, the new interactive host/event actions are fun to play with. You can provide parameters to your scripts via dropdown or a free text field. Here’s a dropdown example. Well, a mockup, because my Python script is currently just a Hello world always returning that it changed the light color. Anyway, will modify my existing lights on/off script to handle colors, too.
So, if in scripts I click on Advanced Configuration, I get to adjust the input type and dropdown options.
… which gives me this:
Now, when I click on Change home office light color, I get to see:
And after choosing any of the colors, I get:
Easy! Just pass {MANUALINPUT} macro for your script as a parameter and it works. Like this:
Will definitely be helpful in serious business applications, as your on-call guys could, for example, trigger any Ansible playbooks through Zabbix to investigate and/or fix something just by clicking on an alert.
DNS monitoring gone overkill
With the new and improved net.dns.get Zabbix agent item key, you can query no less than 73 different DNS record types. To visualize this, your DNS monitoring could look this wild. No, whatsuphome.fi doesn’t give you back answers for nearly all of the query types but at least Zabbix tries.
Next page:
So if there’s something really deep you want to know about your DNS, Zabbix now supports it.
… and much more!
I’ll have lots of poking to do, including creating my custom widgets. But, from now on, bye-bye Zabbix 6.4. Here at What’s up, home?, it’s now time to move on. Oh, and by the way, Grafana also continues working just fine with Zabbix 7.0beta1, or at least I haven’t seen any broken dashboards yet.
This post was originally published on the author’s page.
As AdGuard Home has an API, monitoring it with Zabbix is trivial.
Communicate with the API
Communicating with AdGuard Home API is easy: pass it Authorisation: Basic XXXXXXXXXXXX header, where XXXXXXXXXX is just a Base64 hash of your AdGuard username and password. You can generate that Base64 snippet with for example
echo -n "myuser:mypassword" | base64
Next, in Zabbix, create a new HTTP Agent type item, and point it to your AdGuard Home instance.
Create some items
You’ll get the info back as JSON, so next you can create some dependent items and start monitoring. I only added
Total number of DNS requests
Blocked # of DNS requests
Redirects to safe search
Parental advisory stuff
Average request processing time
For the dependent items, you’ll then just do some JSONPath processing.
Add triggers
Next, I added a few triggers to alert me if AdGuard starts to run slower than usual.
Add service
Finally, I added AdGuard as a new business service, so I’ll get an SLA for it.
And that’s it! From now on I’ll know more about how well my home router ad-blocker is working. (Well, it also has a Skynet firewall which probably filters stuff before AdGuard Home, but that’s another story….)
This post was originally published on the author’s page.
As my home monitoring experiment has become such a celebrity and as it has so much going on, I’m trying to make sure I won’t ever lose its configuration and data, no matter what would happen. Here are some tips and reminders for you, too.
In the IT world, so much can go wrong. Hardware can die, files can become corrupt, malware can hit you, hackers can breach your systems, buggy software updates can cause havoc, you can fat-finger some commands or click on the wrong place and remove data… the list is endless. Here are some ways I’m attempting to protect my home monitoring environment.
Keep software updated
First things first. In today’s malicious world, it’s mandatory that you keep your software updated. With Linux and Zabbix, you don’t have an excuse to skip the updates. Updating your systems is fast and trouble-free. In small environments like home, even major Zabbix upgrades are fast as the database is not very big, so the database migrations that in a corporate environment can take time, will go through in minutes if not faster.
But I don’t trust one environment. What if my BackupPC says Kaboom? A nightly cron job also copies backup archives from my Raspberry Pi to my Mac, which in turn mirrors the backups to my iCloud. And, there’s one more cloud service I’m using for all my backups but not mentioning it here by name.
Test your backups
As long as you have not verified that your backups do actually work, you do not have a working backup. Have a virtual machine into which you can try to restore your backups. See if they work. Test them periodically, either manually or figure out an automated way to do that.
In the case of Zabbix, you can make your primary Zabbix monitor your test environment, and make Zabbix alert if your restore environment Zabbix suddenly starts responding back something else than the regular login page, or if the restore environment database doesn’t come back with some query response you would expect.
Setup a HA cluster
As any hardware can die, it’s not a bad idea to set up a HA cluster. Last winter I was preparing for potential electricity blackouts here in Finland and did setup my laptop to be a secondary node for my Zabbix. This setup has been working very well.
Use strong passwords
Even if it would only be your sandbox environment where you do test new stuff, please remember to use strong passwords. An evil actor can attempt to breach more targets in your network through a single point of failure.
Use ssh keys
Instead of username + password combination, use ssh keys for ssh authentication. Keys are immune to brute-force attempts and with tuning, ssh keys can also be allowed to only connect from specific IPs and run only specific commands. You know how in your ~/.ssh/authorized_keys the lines do start with something like
Obviously, this grants you much more security than traditional logins.
Use HashiCorp Vault
OK, I admit I’m not doing this at home as it would be overkill for my few logins. But, in a larger environment with absolutely critical safety requirements, use HashiCorp Vault for protecting your credentials. Zabbix has native support for it.
Monitor your logs
Setup a centralized log server — it can be your Zabbix server environment, too — and make sure you monitor the logs. My Zabbix gets all my logs, but wouldn’t be a bad idea to use more advanced log monitoring tools, too. Since I already do have ElastiFlow running at home, at some point I might start utilizing Elasticsearch for the logs. Not doing it much yet.
Use chkrootkit, AIDE, others
Tools like AIDE or chkrootkit can help you detect an intrusion. Set them to run in your cron and get alerted in case of any anomalies. Maybe I’ll one day integrate Zabbix with these tools.
Firewall your environment, use VPN
Don’t allow direct access from the Internet to your Zabbix, or your database, or anything really. In my case, my Asus router allows setting up OpenVPN connections, so that’s what I use. Whilst on the go, I just connect to OpenVPN on my phone and do whatever I need remotely through that.
Anything else?
Did I miss something? Let me know in the comments.
This post was originally published on the author’s page.
Can you verify if your phone battery drains faster during summer compared to other seasons with Zabbix? Of course, you can!
I have always felt that during summer my phone battery drains faster than during the darker seasons. This would only be logical, as during summer the phone display must be brighter than during the darker seasons. Additionally, during summer we also tend to be more outdoors, so instead of home Wi-Fi the phone is using mobile data. On the other hand, during winter time if you are outdoors, the cold weather might affect battery life as well.
But how severe is the drainage difference? Let’s check it out!
Analyzing the data
In June, my iPhone battery average level has been 51%
In April, it was around 67%.
In March, about 71%.
What does this prove?
Well, necessarily not much as this is not a very scientific method. However, my day-to-day phone usage does not vary much — if anything, during March/April the drainage should have been worse, as I have a tendency to participate in one daily afternoon meeting from outdoors; I’ll stroll around with our baby and get some fresh air whilst being online in the meeting. Now in June, I’ve been enjoying my summer holiday. But clearly, something is going on as the average percentage difference is such high. For my Apple Watch, the difference is there but not that significant.
Why am I not comparing the data with the iPhone activity? Heh — to prove my point with my earlier “Things that WILL go wrong when monitoring IoT devices” post; something has happened as my Home Assistant won’t provide that information anymore. The same happened with connection-type data. I’ll have to take a look at that someday, but lately, I’ve been busy as a bee with 1) the summer holiday and 2) preparing material for Zabbix Summit 2023.
Real-world applications for this kind of cherry-picking
Even though this post is very thin in its contents, I’m posting this as a hint for you on how you can utilize Zabbix in more serious monitoring targets in this way. Need to compare UPS battery depletion rate? Disk space usage rate? CPU usage? Concurrent connections? Whatever the data, many times it’s useful to stop for a moment and compare how things were (some time units) ago.
Yes, you can simply pick a longer time period in Zabbix time picker and see the data for one year or whatever, and most of the time it will show you a change in pattern, but if the changes are more subtle or the graph is very busy, sometimes zooming in to shorter time periods in history will show you something that you might otherwise miss.
For example, if I expand the time range for the battery usage, not only you’ll notice that the Home Assistant iCloud ride has been a bumpy one, but also the details do get lost when the time range is longer.
More ways to compare historical data
I didn’t build any new graphs or dashboards for this quick experiment, I merely used the time picker. In case one would need to have comparison data available at any time, using time shift and additional data sets would help you out. And, like many of us do, one way to dive deeper into data collected by Zabbix would be to analyze it in Grafana, but that’s a story for another day.
I have been working at Forcepoint since 2014 and I’m happy that my work won’t drain my personal battery. — Janne Pikkarainen
This post was originally published on the author’s page.
When reading this blog, you could easily think that everything is smooth sailing all the time. No. When you monitor your home IoT — or frankly, just USE your home IoT — you have plenty of small details to watch out for. I list them for you, so you don’t have to find them out the hard way like I’ve done over this 1+ year of journey.
1. The status is not what it seems
This is especially true with the IoT devices operating on 433 MHz radio frequency. Your home smart hub sends the signal like a radio station hoping for your IoT device to catch it, and to my understanding, it does not get a reply back from the device. If anything is interfering with the signal, your device will miss the signal and thus your home smart hub will be showing the wrong status.
So, you will need to either get rid of these devices and replace them with devices that use a two-way communication protocol such as ZigBee or if that’s not possible, to set up extra monitoring to try to guess if the command your home smart hub sent actually went through. Did you attempt to power on/off a smart power socket connected to a radiator? Keep an eye on the smart temperature meter and react soon if the temperature does not start to rise after the power socket got powered on, or so.
2. Battery-low messages can be deceiving
Two of my Philips Hue motion sensors have been complaining about low battery status for about six months now, but they are still operating just fine. I’ll let you know when I finally have to replace the batteries on them.
On the contrary, the batteries on some 433 MHz frequency Telldus thermometers can just die without too much warning. For them, your monitoring need to react fast if the values are not coming in. To make things more complicated, not TOO fast though, as sometimes these thermometers can hibernate for some time before reporting new values; possibly when there’s no change in temperature, they will enter some power save mode or something. I don’t know.
3. Bluetooth devices and 2.4 GHz Wi-Fi can interfere with each other
Even though my devices do not use 2.4 GHz Wi-Fi too much, I have some devices like Sonos smart speaker where it’s a must. So, for example, when playing music through that speaker, it’s possible that my Raspberry Pi 4 cannot hear the RuuviTag environmental sensor very reliably. It did help somewhat when I found out that on my Asus router, it was possible to enable some kind of “Bluetooth coexistence” mode, but it’s not a 100% solution for my issue.
4. Make sure any helper components are really up
Along with Zabbix and Grafana, my Raspberry Pi 4 runs Home Assistant to harvest some values about my iPhone and so on. It runs as a Docker image and generally is stable, but sometimes it just stops working. I have an automatic daily restart of that Docker image and so far that has been a relatively good way to keep the image running.
5. APIs can and will change
Monitoring something over some API? Or through web scenarios? Rest assured that your joy won’t last forever. This is IT, and things just won’t remain the same. SOMETHING is guaranteed to change every now and then and the more your monitoring relies on 3rd party things, the less you can trust that your monitoring just would keep on working. No, it’s likely you will need to alter things every now and then.
6. Monitor your monitoring
Even though Raspberry Pi 4 and Zabbix are very reliable and are very unlikely to cause you any trouble, of course, they can fail, or more likely something else will not be like it should. Your home router or Internet connection can die. Electricity can go down. Hardware can die. If you want to be really sure, monitor your monitoring from outside somehow. Have a separate monitoring running on the cloud somehow.
In our case, the electricity and ISP are very reliable, and Cozify smart home hub has a nice feature where the Cozify cloud will text me if the hub loses connectivity — that’s usually a good indication that either the ISP or power went down. Also, I’m about to roll out a small cron job on this site which would check if my Zabbix has updated a test file in a while. If not, it would indicate my Zabbix would be down or otherwise unreachable, so then whatsuphome.fi could e-mail me.
7. You will get paranoid
With more knowledge comes more pain. With some devices, you’ll start to think that they are going to break soon. As an example, the freezer I keep referring to — sometimes it has short periods of time when its temperature for some reason rises a bit and then it goes down again. I don’t know if that has something to do with the fact that our freezer is one of those which does not form ice everywhere so it’s maintenance-free, or if that’s something else, but we keep observing spikes like this about once a week.
I have been working at Forcepoint since 2014 and have learnt not to trust the technology. — Janne Pikkarainen
This post was originally published on the author’s page.
Can you monitor a USB-connected printer with Zabbix? Of course, you can! But can ChatGPT help set up the monitoring? Well… erm… maybe! By day, I am a Lead Site Reliability Engineer in a global cyber security company, Forcepoint. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them. Welcome to my blog about the project.
When it comes to printing, I am not laser-sharp. That’s simply because I have not printed anything in a long, long time, and even if I have, it’s been a printer maintained by Someone Else. Yes, I know how to add paper and how to do a ritual dance whilst printing to prevent the printer from doing an annoying paper jam. Yes, I have added some printer servers under Zabbix monitoring at the office. That’s about where my printer wisdom ends.
Nobody buys a printer anymore, except for us
It’s 2023, and nobody prints anymore, or that’s my personal impression. Well, that changed at our home as we enter another domain I understand 0% about. Occasionally my wife is making some clothes for our baby, me, and herself. For that, she’s printing out the source code… the CAD models… well… the blueprints for the clothes. I kid you not, the clothes schematics look so complicated to me and the text-based instructions so alien that I’m sure the instructions are not from this planet.
Anyway, my wife found a used HP LaserJet Pro MFP M28a for us for a steal. No, not a literal steal, but for 40 euros, which did sound cheap to me, so we bought it. Of course, normal human beings would just connect the printer to their laptop/desktop and be done with it. However, you know how this story continues.
Flirting with ChatGPT
I went to Zabbix integrations and was sure I would be able to easily find out how to monitor a USB-connected printer. I’ll just search and… what? Nothing? No worries! We have the community templates! I’ll just quickly grab a template from there…. the empty result set, AGAIN? Monitoring SNMP-enabled printers is a different story, for that Zabbix can suit you very well, but just try to find out something for USB-connected printers. I know I could probably monitor my printer by spying cups but I’m not sure if it could return the details I wanted.
With my experiments, I’m used to the fact that I cannot get direct answers to my monitoring needs using search engines, as I’m doing stuff that not too many have done. Even if they have, they might not have blogged about it. That’s ok, but as this was about printing which is a common topic, I decided to try out what happens if I try out ChatGPT. I did the tests with the free ChatGPT 3.5, the new ChatGPT 4 would probably be a different ride.
Here’s our lovely little chat.
Me: I have an HP LaserJet MFP M28a monochrome laser printer connected to USB. How do I get its remaining toner level through the command line on Ubuntu 22.04?
ChatGPT: You can get the remaining toner level of your HP LaserJet MFP M28a monochrome laser printer through command line on Ubuntu 22.04 by following these steps:
Install the “hplip” package by running the following command in the terminal:
sudo apt-get install hplip
Once the installation is complete, run the following command to set up your printer:
sudo hp-setup -i
Follow the instructions provided by the setup wizard to configure your printer.
After the printer is set up, run the following command to get the remaining toner level:
hp-systray -x
This will launch the HP Device Manager, which displays the toner level for your printer.
Alternatively, you can also run the following command to get the toner level directly in the terminal:
hp-levels -n
This will display the toner level in a percentage format.
Me: None of those worked as hp-levels and hp-systray require QT4 and D-Bus. Let’s use ipptool instead.
ChatGPT: Sure, we can use the ipptool command to get the remaining toner level of your HP LaserJet MFP M28a monochrome laser printer on Ubuntu 22.04.
Here are the steps to follow:
Make sure your printer is connected to the computer and turned on.
Open the terminal on your Ubuntu 22.04 machine.
Run the following command to install the CUPS and IPP utilities:
sudo apt-get install cups-ipp-utils
Run the following command to find the printer’s URI:
lpinfo -v
The output should contain a line with the URI of your printer, which will look something like this:
ipp://HP_LaserJet_MFP_M28a.local:631/ipp/print
Run the following command to get the remaining toner level:
That’s it! This is how you can use the ipptool command to get the remaining toner level of your HP LaserJet MFP M28a monochrome laser printer on Ubuntu 22.04.
* * *
The latest tip really did work, so we’re off to a very good start with ChatGPT! Next, I tried asking if it could create a Zabbix template for me, but even though it really tried, it did fail.
Adding printer to Zabbix
So, before continuing with the failed ChatGPT attempt, let’s add this printer to monitoring, as that’s the reason you are reading this blog anyway. I was very surprised about the detail level of the ipptool output: everything from supported paper sizes to printing options to other usual details, but then also some dynamic details like printer uptime.
I wanted to add those dynamic details, so I read through the output and added the details in the way you already know me doing:
Zabbix then reads that text file to a master item, and with dependent items and item preprocessing cherry-picks the interesting details
In screenshots, like this.
Does it work?
Of course, it does, as this routine is what I’ve done so many times before in this blog. Here’s the result. And yes, the toner will likely run out soon — this is a used printer and it’s complaining about low toner level every time we print. An interesting experiment to see how many pages we can still print before it actually runs out of juice. For “0% left”, as also reported by other tools, the printer does an excellent job.
Back to ChatGPT we go
If I would copy-paste my complete chat with ChatGPT, this blog post would become ridiculously long. Communicating with ChatGPT was like with a hyper-active intern who proceeds to do SOMETHING only to realize moments later that whatever it did was completely else than what you asked for. Probably I’m just a sucky ChatGPT prompter.
To get you an idea of how ChatGPT failed, here’s a summary of how it failed:
It really attempted to create YAML-based template files for me.
Unfortunately, when attempting to import the templates to Zabbix, that part failed every time
When I told the error messages to ChatGPT, it attempted to fix its errors, but in weird ways. Sometimes it changed the template in drastic ways, even if it was supposed to only add or modify a single line of it. Multiple times, it decided to change the format from YAML to XML unless I demanded it to stay on YAML
Here are a few snippets from the chat. Maybe at some point, I’ll throw in some money to try out ChatGPT 4.
… this went on and on until I gave up. In conclusion: this time ChatGPT nudged me in to correct direction to get the desired output about the printer info (although a sharp-eyed reader might notice I hinted it about the tool I’d like to use and how I might after all know more about printers than I pretended during this blog post…). Then, ChatGPT ran out of ink when it tried to generate the Zabbix templates. It’s scary advanced anyway, and someday I will try out the more advanced ChatGPT 4.
I have been working at Forcepoint since 2014 and luckily I don’t suffer about the empty paper syndrome very often. — Janne Pikkarainen
This post was originally published on the author’s page.
Can you monitor your mobile data usage with Zabbix? Of course, you can! By day, I am a Lead Site Reliability Engineer in a global cyber security company Forcepoint. By night, I monitor my home with Zabbix & Grafana Labs and do some weird experiments with them. Welcome to my blog about this project.
As it is Easter (the original blog post was published two months ago), this entry is a bit short but as I was remoting into my home systems over VPN and my phone, I got this blog post idea.
When on the go, I tend to stay connected to my home network over VPN. Or rather, an iOS Shortcut pretty much runs OpenVPN home profile for me whenever I exit my home.
My Zabbix collects statistics from my home router over SNMP, and as usual, the data includes per-port traffic statistics. VPN clients are shown as tunnel interfaces so Zabbix LLD picks them up nicely.
So, as a result, I get to see how much traffic my phone consumes whenever I’m using a mobile VPN. Here are seven-day example screenshots from ZBX Viewer mobile app.
VPN connection bits sent.
VPN connection bits received
So, from this data I could get all the statistics I need. And, using my ElastiFlow setup, could likely see to where my phone has been connecting to most.
This post was originally published on the author’s page.
I’m entering a whole new level of monitoring and “What’s up, home?” could now also be called “What’s up, me?”. Recently my colleague did hint to me about Home Assistant’s HomeKit Controller integration just to get my HomeKit-compatible Netatmo environmental monitoring device to get to return value back to Zabbix without my Siri kludge. One thing lead to another and now I’m monitoring my iPhone and Apple Watch — so, practically monitoring myself.
But how to get to this level? Let’s rewind a bit.
Home Assistant
Home Assistant is a nice home automation software. It is open source and provides many, many integrations for automating your home. I now have my Netatmo comfortably monitored through that…
Bye-bye, mobile app and my Siri kludge. This screenshot is from Home Assistant.
… but while exploring Home Assistant’s integrations, I came upon its iCloud integration. Oh boy. This takes my monitoring to a whole new level.
But how to get this data to Zabbix?
On Home Assistant, you can go to your account settings and create a Long-lived access token. With that, you then just pass the authorization bearer as part of your HTTP request and you are done. So, like this.
This way you’ll receive your Home Assistant data back in JSON format. As the output is really really really long, and I needed just a relatively small set of data for myself, I cherry-picked those using the above item as the master item and then created a bunch of dependent items.
… and here’s a single item so you get the idea.
Let’s create some dashboards
Now that I have my data in Zabbix, it’s time to create some dashboards. Fascinating that I can now truly monitor my iPhone and Apple Watch like this.
I also created a Grafana dashboard.
Observations
This has been now running for roughly a day for me. Already some observations:
While driving, at traffic lights I tried to see what would happen if I disable the Bluetooth connection between my car and my iDevices. My status was reported as Cycling instead of Automotive for the rest of the trip. Hmm.
Not all the data will be updated in real-time, but there’s a significant lag. Also, it seems I might need to VPN to my home so the data would be updated sooner while I’m not at home.
iPhone’s custom focus modes are not updated to Home Assistant. During the sleep focus mode, the focus mode was reported as On, but for any other mode I tried it only shows Off. Shame, I would have loved to start tracking things like how long it takes for me to put our baby to sleep or how much of the time I’m spending with this blog. That has to wait for now.
But anyway, this thing just opened a whole new Pandora’s box for me to explore.
This post was originally published on the author’s page.
Can you create a proper weather dashboard with Zabbix? Of course, you can! By day, I am a Lead Site Reliability Engineer in a global cyber security company. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them.
Since Zabbix 6.0 has provided you an official OpenWeatherMap template. It gives you all the standard weather details: temperature, humidity, and so on, for any location you’d like to observe.
However, by default, Zabbix does not come with a weather map template. Can we add one? Probably. I say probably because it looks like the free OpenWeatherMap account I created might not have enough credentials to show the layers. Still, let’s give this a try!
(And before you ask why I did not just add another custom geomap provider under Zabbix Administration –> General –> Geographical maps — I wanted to have these layer toggles for clouds etc, and that requires custom JavaScript)
Getting OpenWeatherMap account
Just go to the OpenWeatherMap site and create an account for yourself. Soon enough, you’ll get an API key you are supposed to use.
Embedding OpenWeatherMap to Zabbix UI
I found leaflet-openweathermap and even though it’s abandoned, the demo that comes with it seems to work just fine. Embedding that to Zabbix was not that of a big deal.
Clone the git project for yourself
Copy the example somewhere where you can serve it, I did put it on my Zabbix server under /assets/openweathermap/ directory
Load that map in an empty tab to verify you see it works for you
With the default App ID that is bundled with the map the layers do work, but it would not be cool to use the author’s API key as stated in the code
Change the AppID to one you have received … well, at this point it stopped working for me, but if you really need it, OpenWeatherMap is not that expensive
Then you can add it to Zabbix just by inserting a new URL widget and pointing that to your location.
How does it look like?
Here we go! And, as another idea for you, with the URL widget, you can embed any camera input to your dashboard, too, some hints in part 21 of this blog I don’t want to show you our own camera footage, so I added Lauttasaari, Helsinki location — that is where Forcepoint has its office.
Now that’s a weather dashboard for you.
Get alerted
OpenWeatherMap would also support alerts about severe weather conditions, but another option would be to find out your local weather data from your nearest provider and use their open data for this; in Finland for example, Finnish Meteorological Institute has its own open data for one to use. Then just add those to Zabbix via HTTP Agent item type for example much like I did in part 32 of this blog, and you’re done.
I have been working at Forcepoint since 2014 and as you know by now, I have this never-ending drive for monitoring. — Janne Pikkarainen
This post was originally published on the author’s page.
Can you monitor how much you drive your car, even if your car wouldn’t have any way to report back to Zabbix? Of course, you can! By day, I am a Lead Site Reliability Engineer in a global cyber security company. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them. Welcome to my blog about the project.
Some forewords: Now that our baby girl is over six months old, she has developed some kind of sleeping pattern. It means she goes to bed very early in the evening, around 6pm. Or, I go to the bedroom with her and wait for her to sleep steadily before I exit the bedroom without her noticing. It means I have lots of time to think, and also to play around with apps like iPhone Shortcuts. I have previously done a few Siri & Zabbix experiments and this will be one more.
I did do this shortcut only two days ago and have not actually driven yet, but I verified that the shortcut itself works when I go into my car and start it up. Also, as I don’t want to give out the exact location where we live, for this blog post I faked our car to be located in Santa Claus Village, Rovaniemi.
Let’s get started.
What are you planning?
Even though I already know very well how much I drive — there’s the odometer in our car, a fuel app in my iPhone shows how many liters per month I refuel, and so on, this data is still something that would contribute to my dear single pane of glass, like your company probably wants to have.
My Siri Shortcut is simple: whenever I go to my car and my iPhone connects to car Bluetooth, it’s a clear data point that I’m probably going somewhere, so the shortcut gets my current location and saves its coordinates to a text file in my iCloud.
Next, just like in my previous Siri examples, a Zabbix Agent on my MacBook keeps an eye on this text file, very much like in my FlightGear integration example, Zabbix will then populate the coordinates in Zabbix inventory for my car host. This way, I can project the car location to the Geomap widget.
Let’s create the shortcut
Here’s the shortcut in all its simplicity.
About that Append to Text File… why appending instead of overwriting, I’ll tell you a story some other day.
Why Desktop Directory? I’ll tell you a story some other day.
Next up, Zabbix
On the Zabbix side of the house, the story is like so many times in my posts: read the text file, and using dependent items create the longitude and latitude items.
Wait! You saved it on your Desktop but now it’s in /tmp? I’ll tell you a story about this kludge some other day… or immediately after this caption.
It was easier to get macOS Zabbix Agent to get to read /tmp instead of your home directory, as the security is in the way, so a cronjob syncs the file once per minute to /tmp. Not only that but because in iOS Shortcuts the Append to a text file was the only way I got the shortcut to run without it asking for permission to run, my cronjob is actually like this:
Beautiful? No, but due to reasons I had to do this, and at least it works.
Anyway, then the longitude/latitude-dependent items just use some regular expressions.
Beautiful? No, but it works.
Does it work?
Of course, it does! See for yourself.
Here’s the latest data…
… and here’s the Geomap.
But wait! How does this track your kilometers?
Heh, you got me. It does not. One easy way would be to use Get distance block in iOS Shortcuts. It actually works — you get to choose that yes I will be driving, give me the kilometers. Whenever I do that, I would need a text file containing just one line (which would contain the old location), and getting to that point without your iPhone asking anything ever is not so simple, so for now I gave up.
So, the next part of this will be to use some API and make my Zabbix calculate the distances. That would be cooler anyway, but I’ll find time for that next time. Anyway, from now on Zabbix will know the locations where I have started our car, so the data will be collected from today. I know there are limitations in this implementation, such as that if I start the car and just drive to some place and back without ever stopping the engine, that won’t really give me any results, but this is better than nothing.
I have been working at Forcepoint since 2014 and as you know by now, I have this never-ending drive for monitoring. — Janne Pikkarainen
This post was originally published on the author’s page.
Can you monitor your website visitor rate with Zabbix? Of course, you can! By day, I am a lead site reliability engineer in a global cyber security company. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them.
I have this website hosted in a domain hotel, and among other features, the admin panel has some standard website access log analyzers (such as awstats) available for me to see the activity of this site. That’s cool, but also boringly easy, and requires me to log in to that admin panel instead of me using my trusted single pane of glass that is Zabbix.
Let’s connect to site logs
If I log in to my site over ssh/sftp, my home directory has a preconfigured access_logs directory. Like the name says, it contains the website access logs in the usual format you would expect it to be:
That’s great, but how to monitor that in real time with Zabbix? Let’s use sshfs — it’s like NFS or CIFS, but allows you to mount stuff over ssh. On my dear Raspberry Pi 4, which runs my Zabbix, running
did mount my remote server access_logs directory perfectly fine.
Time for monitoring
Now that we have our log file, the rest is very straightforward and standard log file monitoring. First, let’s add a master item that reads the log.
Nothing too difficult yet.
Next, let’s add a dependent item that grabs the visitor IP address part from a log line.
… and some item preprocessing to grab only the IP
Items
Sorry about that ugly regular expression.
… after adding a few more items, here’s my template.
I’m currently not parsing the referrer, exact URL, or user-agent values, as for the most part those would just add unnecessary noise and load for my poor little home Zabbix.
Dashboard time!
So, finally, I created a dashboard showing the number of unique IP addresses & hits during the past 24 hours and some graphs. Now that I’ve not posted any posts in a while, welcome to Tumbleweedville!
It’s so silent in here that I can hear my own typing.
After publishing this post, I’ll wait for a while and then update the post with a new screenshot, so we’ll get to see the incredible visitor surge that will be counted in at least tens of new IP addresses.
Update #1 about 15 minutes after publishing the post
Clearly some movement in the access log needle!
Update #2 about 15 minutes after publishing the post
Almost 400 unique IP addresses already? Hello, dear readers and bots.
Update #3 about 15 minutes after publishing the post
Even though IP addresses are a bad way to measure the actual amount of visitors, roughly 400 unique new addresses after publishing my post are very good. Thanks, bots and readers!
This post was originally published on the author’s page.
Is Raspberry Pi 4 a goodbye or a good buy for running Zabbix? How is it performance-wise? Is it reliable? Here’s my nine-month review of it, with a splash of appliance/application performance monitoring.
In about April 2022 when it became clear that I am going to continue my home monitoring project, I bought a Raspberry Pi 4 to run the show. Here’s my opinion on how well it is suited for running Zabbix.
Installing Zabbix
Applying that delicious layer of Zabbix on top of your Raspberry Pi 4 cake is extremely straightforward, as just like for every other platform that Zabbix officially supports, they do have packages and instructions to set up what you’d like to run.
So many options to choose from!
After installing the packages, the next steps are just like with Zabbix running on any other platform, so I am not going to dive into that now.
Modifications to my Raspberry setup
As I do not need to run a graphical environment on my Raspberry, I did disable the graphical environment from starting at all to save some precious RAM and other resources.
After some time I did also purchase an external USB hard disk, as the memory card from where Raspberry Pi 4 runs its OS is not very snappy, especially with write operations, and can also run tight on free space.
Other than that, my Raspberry Pi 4 is running pretty much by default.
How about the performance?
The graphs that you are about to see are from nine months period of time, as that’s about as long I have had the device.
No problem with the CPU usage. It’s been creeping up a little bit over time though, as I have been adding new items to monitoring and also additional software, such as HomeBridge and Home Assistant.
It still has available memory, even though the device runs Zabbix server, MariaDB, Grafana, Mosquitto, Home Assistant and HomeBridge.
As you can see, the number of running processes has grown significantly as I have been adding other stuff than Zabbix.
It’s easy to see when I did switch from an internal memory card to an external USB drive. The disk I/O utilization percentage is hovering at very tolerable levels.
I/O latency has remained about the same.
With only Zabbix, MariaDB and Grafana running the device remained around the 55-60C area, but has been warming to about 70C with the additional software. Still not too bad.
Splash of APM
Have you ever wondered what happens to the memory usage of a wrapper shell script that runs other scripts in a loop and keeps doing that until it’s manually stopped? This happens, it’s boringly stable. The results are brought you to by Zabbix Agent 2 process discovery.
Really, it does not vary much.
But as I have been adding new stuff, clearly the OS needs to do some more swapping and even the script has more page faults than before.
There’s more than that to process discovery, but those were some examples.
Zabbix server itself is doing very well, here are some example stats.
My conclusion: Raspberry Pi 4 is an excellent Zabbix server for smaller environments and a very good Zabbix proxy candidate. It’s been rock solid.
This post was originally published on the author’s page.
Can you follow the news with Zabbix? Of course, you can! By day, I am a lead site reliability engineer at a global cyber security company. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them. Welcome to my blog about the project.
A long time ago, before the dawn of social media, RSS (Really Simple Syndication) readers were all the rave. Instead of visiting each site you followed individually, you could add their RSS feeds to your RSS reader, which then would show you the latest news titles from as many sources as you wanted. Not only the titles, but depending on the news site you could also read a teaser or even the full news through your RSS reader without ever visiting the site itself.
Is RSS still a thing?
This was all good for the end-users, but the beancounters at the news companies got worried, as of course without visits to news sites, the advertisement income would come down, too. RSS readers can still be useful, but…. oh, I’ll need to stop, this is not the scope of this blog post.
Instead, the underlying technology of RSS is what makes it interesting. It’s just XML, so easy to consume by whatever software. Even though RSS is not a media darling anymore, it’s useful for gathering info from various sources to be then utilized somewhere else — like in Zabbix.
Let’s follow this site
So, how to follow the latest posts on this site through Zabbix? Easy, as this is just about parsing some XML.
Let’s begin with adding a new HTTP agent item.
With that in place, let’s add some dependent items, with the end result being this:
Each of those is just dependent items with some item pre-processing — the example below parses the first occurrence of title to the text.
How to use this?
In this case, I created a separate dashboard to show the latest blog post title, a link to it and the publication date. Wouldn’t be too hard to create a custom Zabbix module to make this fancier, but let’s leave it for another day. For now, by just using Item value widget types, we get this.
In the real world, there are plenty of actual use cases. Use it to alert you about the latest vulnerabilities, updates or other news about the stuff you have in your environment. Create a news dashboard for your security operations team or developers. If your own products do utilize RSS for something, this also can be very handy for end-to-end testing, as both Zabbix and your eyes can spot visually if something is not right. I’m sure you can come up with more and better ideas.
I have been working at Forcepoint since 2014 and never get tired of the news. — Janne Pikkarainen
This post was originally published on the author’s page.
Can you detect a crying baby with Zabbix? Of course, you can! By day, I am a monitoring tech lead in a global cyber security company. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them. Welcome to my blog about the project.
Time really flies. Our little baby girl at home is already about three and a half months old, and that shows in so many ways. If during her first month or two she cried quite a lot and quite easily due gassy stomach and whatnot, she nowadays mostly is a chill mini-human just observing the world.
Which then raised the question for me — how often she cries? Could I monitor that? Oh yes. And oh no. We’ll get to no part later, but let’s start with the good bits.
Hey Siri, help me
As I pretty much always have my iPhone with or near me, and wear my Apple Watch nearly 24×7, I thought I would give their sound recognition abilities a try.
To start, I opened the Settings on my iPhone and went to Accessibility → Sound recognition → Baby crying and enabled that.
Next, I opened Shortcuts and created a new Personal automation.
Here’s the advanced CRYENGINE in action.
So, every time my iDevice thinks that our baby is crying, it appends to a text file stored on my iCloud account.
Zabbix Time!
How to get that data to Zabbix?
I have a MacBook Pro and a Zabbix agent running on that, so the next natural step was to make it monitor that particular text file. There would be so many ways to detect if this file has been changed; as I’m appending to the text file, I just made Zabbix keep an eye on the file size.
How does that look like on my graphs? Not so surprisingly, the value changes.
I also set up a simple trigger that screams if the file size has changed since the last check.
The result? Well, here’s some alert history.
Sleep Learn Adapt reporting
I also added our baby as a Service to my Zabbix, just because it was too fun to skip as the terminology involves child services, parent services and such.
And here’s a totally inaccurate and unmeaningful SLA report about her.
The wobbly bits
Just like our baby is still clumsy and has a lot to learn, it seems that Siri is like that too when it comes to detecting emotions. Yes, Siri can detect if our baby is crying, but it also easily gets worried whenever our baby makes loud joyful sounds. It reminds me of the golden times when T-800 practiced smiling. It just doesn’t know or understand human feelings. At least, not yet.
Then the other odd part is that about one day after I enabled the cry detection on my iPhone, it’s not doing it anymore. Did the detection process crash? Probably. I have not restarted my iPhone yet, because I just could not bother to do that yet. But, in theory, we can detect a crying baby, or many more usual sounds like a doorbell, a dog barking just by using Siri and Zabbix.
I have been working at Forcepoint since 2014 and I think my human skills still beat the ones Siri has. — Janne Pikkarainen
This post was originally published on the author’s LinkedIn account.
Can you monitor Northern Lights with Zabbix? Of course, you can! By day, I am a monitoring tech lead in a global cyber security company. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them. Welcome to my blog about this project.
Christmas is coming, and (at least if you believe Hollywood movies) part of that magic would be staring at the sky and marvel the Northern Lights. Well, in practice you probably won’t see them, as even if the Northern Lights would be up there, a thick layer of clouds will probably prevent you from seeing them. Or then you live in an area with so many street lights that you don’t see the sky properly.
We have tried to watch them several times with my wife, but our attempts all over the years and all the seasons have failed so far. But, for the sake of the Christmas spirit, let’s imagine you could actually see the lights.
Getting the data
There are probably actual APIs for getting the data — at first, I went to NASA’s open data site but then quickly gave up; there’s so much data that I would not have an actual idea how to start parsing this beautiful sky flames phenomenon.
Admitting my lameness, I next came up with plan B. The Finnish Institute of meteorology has this page for space weather & Northern Lights predictions. Sorry, the page is all in Finnish, so likely it looks like an alien language to you. Anyway, there’s this snippet that shows the probability of Northern Lights tonight (“Tänä yönä”), tomorrow (“Huomenna” and the day after tomorrow (“Ylihuomenna”).
Is that some kind of advanced form of encryption? No, that’s just the Finnish language for you.
Making it work
But how to parse that? Well, of course, with Zabbix, that is easy with the HTTP Agent item type. It allows you to grab website content and then perform all the advanced processing for the data you would expect from Zabbix item preprocessing.
Then, using dependent items — one for tonight, one for tomorrow, one for the day after tomorrow — and item preprocessing we can extract the interesting bits.
And see, it works!
I also created a (still boring-looking) dashboard, which shows me the current values.
The problem I now have is that I don’t know all the values the page could contain — when I created this blog post, the chances of seeing the Northern Lights were small (“pieni”) or smallish (“pienehkö”). Well, I keep checking my dashboard from now on! For now, I could create triggers that would alert me if the values would be something else than “pieni” or “pienehkö”, but did not have time for that yet.
I have been working at Forcepoint since 2014 and I bring many Nordic values to the company, even though I’m not lucky with the Northern Lights. — Janne Pikkarainen
This post was originally published on the author’s LinkedIn account.
Can you monitor the FlightGear flight simulator with Zabbix? Of course, you can! By day, I am a monitoring tech lead in a global cyber security company. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them. Welcome to my blog about this project.
FlightGear is an awesome free, open-source flight simulator. I am not a pilot, not even a good virtual pilot, in fact, probably the virtual cabin crew would be chanting “BRACE! BRACE! BRACE! HEAD DOWN! STAY DOWN!” to my virtual passengers. Anyway, learning to fly would be awesome.
But what good would be virtual flying without any monitoring? Most people, they wouldn’t care about monitoring. For me, that’s everything I care about with this experiment.
FlightGear Properties
FlightGear can expose all kinds of flight-related data in many different ways; XML logging and via its built-in HTTP server, for example. This time I used its HTTP server, and cherry-picked only a few values (aircraft latitude, longitude, altitude, and speed), as the complete property list is LONG, and I do not understand most of it.
Anyway, you get the FlightGear HTTP server up and running by launching it like
fgfs –httpd=5480
… where 5480 is the port number where HTTP server will be listening on.
You will then have a property browser available on http://localhost:5480/json/ which is from where I found the values I wanted to harvest for my little experiment to see if this thing would fly.
Adding items to Zabbix
To get these values monitored, I added two new master items to Zabbix: one for velocities and one for the position. Then, dependent items are using those master items.
My latitude/longitude items also do populate the Zabbix inventory latitude/longitude fields for my aircraft.
Does it fly?
Yes, it does. I can now have data about my virtual flight.
And thanks to inventory fields, I can show the location of my virtual aircraft on Zabbix geomap.
If you are a flight simulator enthusiast, feel free to use this technique and possibly gather all the values from FlightGear property browser by using low-level discovery. For my little test, I did not bother.
I have been working at Forcepoint since 2014 and have learnt that proper monitoring makes sure your projects do takeoff without too much pain. — Janne Pikkarainen
This post was originally published on the author’s LinkedIn account.
Can you have a Zabbix HA cluster at home? Of course, you can! By day, I am a monitoring tech lead in a global cyber security company. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them. Welcome to my blog about this project.
The winter has come, and due to world events, it might bring one to two hours of rolling blackouts here in Finland, too. As I have my home Zabbix running on my Raspberry Pi, without a UPS this would mean my Zabbix possibly could not monitor the actual duration of the outages, as my Zabbix server would be without power, too, right?
No. Thanks to the simplicity of setting up a HA cluster with Zabbix, I now have a two-node Zabbix server setup at home, with the standby node running on my laptop, which of course can run on battery for the duration of the blackout. So, while this post is kind of boring — I’m not introducing anything weird to monitor today — I hope the post encourages you to try out the high-availability features of Zabbix. It’s easy!
Set up the nodes
As written on Zabbix documentation, setting up HA on Zabbix means two additional lines added to your zabbix_server.conf file:
HANodeName for the descriptive, unique name of the node
NodeAddress, which should be the address Zabbix front-end will then use
That’s it! And, that is what I did. Then make sure your Zabbix servers point to the same database, and that all your Zabbix servers can connect to that database.
But does it work?
Of course, it does! Here’s the status as seen from Zabbix Reports → System Information:
And here’s the status as reported by sudo zabbix_server -R ha_status from the command line on my Raspberry Pi:
Out of curiosity, I tried out what happens if I try the same command on my laptop. This happens:
Still to do
As nowadays due to our baby my time is very limited, I do have one remaining task to make this perfect: to set up a database cluster. For now, MariaDB is running on my Raspberry Pi only, so I would need to spread it to run on my laptop, too. I will most likely do this with MariaDB Galera Cluster, but that will be another story.
Winter, you might take out my electricity, but you won’t take down my Zabbix.
I have been working at Forcepoint since 2014 and I won’t let my systems go down. — Janne Pikkarainen
This post was originally published on the author’s LinkedIn account.
Can you create fairy tales with Zabbix? Of course, you can! By day, I am a monitoring technical lead in a global cyber security company. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them. Welcome to my blog about this project.
We all know how Zabbix has a never-ending list of integrations for just about everything — need to integrate it with OpsGenie, PagerDuty, Teams, Slack, or something else? No problem, there’s probably a ready-made integration for that already.
But, based on questions I’ve received over the years at work, not everyone realizes how utterly powerful the alert message templating engine is for you to create custom messages with the help of built-in macros and of course the user macros you can define. The default Zabbix HTML e-mail message template is very compact in its format, and for me easy to read, but years ago someone at work told me that the alerts were not easy for him to follow.
What I did back then was that I created an alert template of my own, which tells about the events in a bit different format, here’s a short snippet from those alerts.
Fairy tale time!
Now that at home we have our almost-three-months-old-baby, I’m using her as the perfect excuse to make Zabbix alerts to be like fairy tales. You know the drill. Your kiddo wants to hear yet another story before he or she falls asleep, and you have already run out of fresh stories to read.
What if your Zabbix would generate fairy tales for you? Well, not really, but at least the following would make the stories a bit more amusing to you and very confusing to your kid.
Let’s first create a new media type via Zabbix Administration –> Media types. For this, I just cloned the default HTML e-mail media type and gave it a name.
And then, my fantastic story template looks like this:
Add the template to user media type
Next, to actually receive these alerts, you need to configure your user profile and in its media types add the new media type.
Using the template
Getting the new template into use is easy; just go to Zabbix Configuration –> Actions and create a new trigger action with whatever conditions you like.
And then on Operations tab make Zabbix send the alerts via your new fairy tale media type.
The alert e-mail
So this is how the e-mail looks like.
Now go and add some CSS, pictures, whatever you like to your stories. And, perhaps, unlike me, go and change the {ITEM.DESCRIPTION} macro to contain also some instructions what to do with the alert, like at our custom alerts at work I have a tendency to add some hints about how to resolve the issue.
I have been working at Forcepoint since 2014 and I would have many stories to tell you about all these years. — Janne Pikkarainen
This post was originally published on the author’s LinkedIn account.
Can you monitor your backups with Zabbix? Of course, you can — and you should! By day, I am a monitoring tech lead in a global cyber security company. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them. Welcome to my blog about this project.
Some time ago, Zabbix blog had an excellent read about how to back up Zabbix. While I’m next time coming up with something Completely Else in this blog, this week I wanted to remind you about backups.
The topic came into my mind as my old trustworthy Apple Time Capsule from 2013 probably just said “No, I don’t want to do this job anymore”. It’s a good thing that I also regularly back up my Mac to an external USB drive which I only keep connected to the Mac during backups and then put it in a safe place, and that I also do backup to iCloud and one other cloud service. Update one hour after publishing this blog post: Apple Time Capsule came back to life! But, I shall not trust it anymore.
But, for backing up my Raspberry Pi 4 and its Zabbix, I do something different. Let’s move on to that, shall we?
My old and trustworthy friend
A long time ago (20 years ago long time), I used BackupPC at work. Back then I liked it a lot, as it uses proven, field-tested components (Apache, Perl, rsync), it’s fast, light and does its job with no questions asked.
Until lately I took my home Zabbix backups to an external USB drive connected to our home router by letting a cron job to dump the database with mysqldump and copying some custom script directories around. As this What’s up, home? thing is now world-famous and my home Zabbix has turned into an invisible family member, it was time to take backups a bit more seriously — I don’t want to lose my precious home Zabbix configuration.
So, I went to BackupPC’s site and was surprised that it’s still developed! Kind of the latest release is from 2020, and oh boy, it’s much more polished than it was 20 years ago.
Faster than I can say “BA-NA-NA”, I typed sudo apt install backuppc on my spare laptop, which nowadays runs Ubuntu 22.04 LTS.
Meet BackupPC
Here it is! Thanks to rsync and symbolic links, this thing did deduplication long before it became trendy.
And it lets me go back in time via a nice web interface from which I can easily restore individual files, directories or just everything. On October 24th, I did some testing, so don’t scratch your head about that day in the screenshot.
Zabbix, meet BackupPC
At first, I tried to use a Zabbix community template for BackupPC. It worked — for the most part. Unfortunately, at least with a combination of Zabbix 6.2 running on Raspberry Pi 4 and BackupPC 4.4.0 running on Ubuntu 22.04 LTS, the template was unable to fetch metrics, as Zabbix agent’s web.page.get was inserting some garbage between HTTP headers and the expected JSON content — I guess I did hit this issue.
To work around the problem, I just changed the templates master item type to be HTTP Agent instead of Zabbix agent web.page.get. The difference is that with the Zabbix agent-based solution, the agent can fetch the content locally from the BackupPC server so its web interface does not need to listen on anything else than localhost, and now I opened my BackupPC to listen on the network and did let my Zabbix server to fetch the JSON.
If you want to try out my (very) slightly modified template, it’s available on my GitHub.
After the initial hurdles, the rest was a snap. Not every metric is working for me, as I understood that for them to work, I would need to use the bleeding edge version of BackupPC, but for now, this is good enough. I can get all kinds of metrics about BackupPC, below is a custom dashboard I created for it.
… and even if my BackupPC is backing up my Zabbix perfectly now, it is not backing up itself as I did not yet configure it to do so, so here’s how BackupPC would complain about my Zabbix backups too, should they be failing for any reason.
Do not trust your backups
What’s worse than having no backups at all? Doing backups, but ONLY doing backups. Especially if you are just setting up the backup routine for something completely new, it’s very likely that you miss something during the first try.
Backups are not a fire-and-forget thing, you better monitor them and regularly test them, or otherwise, it’s guaranteed that Mr. Murphy will hit you with a clue stick sooner or later.
So, here’s my €0.02:
Take regular backups
Monitor them
Test your backups
Make sure you have more than one backup (my MacBook copies the Zabbix backup files from my Ubuntu daily, too, and from there my Zabbix backups are spread to everywhere my Mac backups are)
My future development in this area includes a script that would attempt to automatically restore the database and my scripts to a VM/container, install Zabbix and see if it works. If it would not, then my Zabbix would get alerted.
I have been working at Forcepoint since 2014 and during my long IT career have learned that you always have to have a guaranteed way to restore something. — Janne Pikkarainen
This post was originally published on the author’s LinkedIn account.
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.
