Can you monitor your Bluetooth headset usage hours with Zabbix? Of course, you can!
By day, I earn living by being a monitoring tech lead in a global cyber security company. By night, I monitor my home with Zabbix and Grafana and do some weird experiments with them. Welcome to my weekly blog about how I monitor my home.
I adjust and tweak myself with the power of music. Finding out the root cause for a severe outage or just fixing some less severe error becomes much more epic if I listen to Hans Zimmer’s music. Trance, drum ‘n bass, demoscene music, and retro gaming music keep me afloat if I have something simple, repetitive things to do. For some reason I write each and every of these home monitoring entries with the soundtrack from the latest Batman movie playing background, and so forth.
My music listening habits, the online meetings at work, and the fact that I mostly work from home, just like my wife, means that I use my Valco headphones several hours a day. Valco claims that their headset can provide about 40 hours runtime with a single charge, and that kind of must be true as I only charge the headset on Sundays for them to be ready for a new week on Monday morning.
But how much do I really use my Valcos? Zabbix to the rescue!
Mac to Valco, Mac to Valco, please respond
As I use my headset mostly with a MacBook, I needed to find out how to get the connection status info from macOS command line. I am sure there are more sophisticated ways of doing this, but the sledgehammer method I used is good enough for my home use.
On macOS, system_profiler command gives you back tons of data, one of the elements being the Bluetooth devices. Sure enough, my Valco headset is visible there, and so is the connection status.
Now that I have the data available, I could send all this text output to Zabbix and use Zabbix item pre-processing. This morning (yes, I created this whole thing only two-three hours ago) I did something else though.
You know, while I was testing if my attempt works in real-time, I created a terrible shell one-liner, which I now also use with Zabbix.
Can you integrate Zabbix with Siri? Of course, you can! By day, I am a monitoring tech lead in a global cyber security company. By night, I monitor my home with Zabbix & Grafana and do some weird experiments with them. Welcome to my weekly blog about the home project.
I have lost count of exactly when, but a couple of major iOS/macOS versions ago Apple’s Siri gained the Shortcuts application. It allows you to automate all kinds of stuff and do some drag-and-drop ‘programming’.
What do I use it for? You guessed it right — I integrate Shortcuts with Zabbix API.
Setting up the Zabbix side
For my home Zabbix environment, I do not have any complex access rights set. So, setting up the API token for Shortcuts to consume was almost a one-click operation. In Zabbix, I went to User settings → API tokens → Create API token and let it do its stuff.
Creating a new shortcut
Now that I have the API token in place, next we need to create the shortcut. That’s not too much work though — run the Shortcuts application and create a new shortcut. What the shortcut below does is:
calls Zabbix API and requests our fridge temperature
parses the value and appends “degrees Celsius” to it
returns the value
Yes, that’s all of it. Drag ‘n drop a couple of elements and assign some values. Done.
Time to get Sirious
Ok, so we have our shortcut in place. What happens if I now ask Siri to check beer temperature? This happens.
The result is actually our refrigerator temperature, the beer thing was just to make this more interesting. But, as you can see, integrating Zabbix with Siri or vice versa is not too hard.
Any real-world use cases for this, other than the geek factor? I don’t know. It might be handy to request the latest alerts or similar from Siri if I’m driving my car and I get to hear that something’s wrong at work.
I have been working at Forcepoint since 2014 and I confess I actually use Siri for some basic work stuff, too. — Janne Pikkarainen
Can you take screenshots of the websites you monitor with Zabbix? Of course, you can! By day, I earn my living by being a monitoring tech lead in a global cyber security company. By night, I monitor my home. Welcome to my weekly blog about how I monitor my home with Zabbix & Grafana and do some weird experiments with them.
Proper full-stack monitoring does cover everything from the device’s physical status to what the end-user sees. However, often even with website monitoring, web admins only rely on all kinds of run-time data; operating system load, hardware health, service response times, and logs. If your site has some rarely occurring technical issue, you might not have any idea what the site looks like when the error happens. Is your site looking really wrong or is maybe just a panel or two misbehaving during some issue?
No, even if this post is meant to provide some food for thought, I am not speaking about the selenium your metabolism needs. Instead, I am speaking about a web-test framework that allows you to automate web browsers by using Java, Python, or several other languages it supports. It’s probably most useful for web browser & website developers so they can be sure that their sites look OK on all major browsers and so forth, but why not utilize that with your monitoring, too?
With a really short Python script (or whatever language) you can spawn a headless instance of whatever browser you want and ask it to do stuff for you. In the horribly simple example above, the script can be called like
and it then saves the screenshot under /usr/share/zabbix/assets/webtest-screenshots/ directory which is accessible by my home Zabbix & Grafana.
Now, where did I put that script?
What I actually use this for at home is that whenever some lunch restaurants publish their new menus, my Zabbix grabs a screenshot of the lunch menus and I can then spy the menus without a need to go to sites by myself.
As I do not want to show the more exact locations of the places I visit, in this example I’ll show you how to screenshot the blog.zabbix.com site instead.
Hooking the script to Zabbix is easy; just add the script to Zabbix via its Administration –> Scripts, and add it as a script you can call via Actions, and if you so want, also put it as a manual action so you can call the script manually any time you want through Zabbix contextual menus.
Then just add a trigger that snaps a screenshot whenever your web test is failing.
Now that we have our screenshot mechanism in place, this is how it looks through Zabbix URL widget:
… and this is how it looks when embedded to Grafana with its Text panel, HTML content type and a simple <img> tag:
What’s this useful for?
Lots of stuff. From now on, you too can easily embed a screenshot of your monitored website within your monitoring environment. Put some more panels to that dashboard showing the active alerts, graphs, run-time data, and logs, and you can have a very comprehensive full stack monitoring in place.
Or make your screenshot script more intelligent (I recommend that anyway), and make it save the files with date stamp info in filenames, so you can have a nice little time machine inside Grafana and its time picker.
Selenium can also give you many more other details, such as performance data or the URLs which were called during the page load. I’m sure that’s useful, too.
I have been working at Forcepoint since 2014 and never get tired of getting a more clear view of the status of the stuff I monitor. — Janne Pikkarainen
Can you monitor a home air conditioner with Zabbix? Of course, you can! By day, I am a monitoring tech lead in a global cyber security company. By night, I monitor my home. Welcome to my weekly blog about how I monitor my home with Zabbix & Grafana and do some weird experiments.
At the moment I was writing this blog post, the summer — and thus maybe the heat wave season — was only approaching. For the past two summers, our home has been a hot place to be. Enough is enough, so that day we got an air conditioner. It’s not a very high-end model, but these things currently come with built-in Wi-Fi.
Wouldn’t it be cool to monitor that with Zabbix? Yes.
Wi-Fi, do you read me?
Getting the air conditioner connected to our Wi-Fi was as easy as the manual promised: press the Health button eight times in a row and wait until the air conditioner says “be-be-be-be-be-beep”. Sure enough, that happened, and moments later the AC Freedom app I installed on my iPhone started to show this.
For a normal person, this would be more than enough. For me, this was only the beginning and the next step would be adding the thing to Zabbix.
Checking the first things first — no, Zabbix does not seem to support this AC out of the box. No worries, that is not the end of the world, it just slightly slows things down, and also makes things a bit more interesting.
Now that the AC was connected to our Wi-Fi, I first went to check out some data about the AC from the Wi-Fi admin interface. It revealed to me that the device contains network hardware by Broadlink in it. Ah-ha! Search engine, here I come!
Okay, MQTT it is. That’s a lightweight protocol designed for IoT device communication, and I had absolutely no clue how that worked, as this was the first time I got to use it. It would blow if this step proved to be too cumbersome.
Luckily, thanks to open source and 2022, getting it to run was not that hard.
It’s nearly summer, welcome mosquitos
The aforementioned Broadlink AC to MQTT quickly raised my confidence, as it immediately found my new device. Yes, I can do this!
… that’s nice, but how to use this any further? I could not see any MQTT messages anywhere.
Soon enough I realized I need to install an MQTT message broker to catch the messages and I found Eclipse Mosquitto.
An apt install mosquitto mosquitto-clients and some config file guessing later my jaw dropped, as I saw this:
Wow, that’s a wind-wind situation. It returns sane values! My next step was then to find out if I can somehow access those URL-like paths with Zabbix.
With Zabbix, MQTT is just a breeze
I remembered from some ancient Zabbix Summit that Zabbix 5.x gained Modbus/MQTT support. My Raspberry Pi 4 is running Zabbix 6.0.4, so certainly that part should be covered.
In the end, getting MQTT to run with Zabbix was almost too easy. Zabbix agent 2 has native MQTT support with its mqtt.get active check, so I tried to add an item like this:
And, as this is Zabbix, of course, it works:
Yay! From now on, my home Zabbix can alert me about the AC as well and generate some fancy graphs.
As I’ve got the AC unit recently and this is just the beginning, I still have more things to add later.
Add some “Yikes! It’s too hot!” triggers
Create a Grafana dashboard
Try out if I can adjust the air conditioner settings using Zabbix
Anyway, in the end, this certainly was easier than I expected.
I have been working at Forcepoint since 2014 and monitoring has never been cooler. — Janne Pikkarainen
Can you integrate Zabbix with a remote control? Of course, you can! Does that make any sense? Maybe. Welcome to my weekly blog about how I monitor my home with Zabbix & Grafana and how I do weird experiments with my setup.
By day, I do monitoring for living in a global cyber security company. By night, I monitor my home. This week I have been mostly doing remote work… gaining physical remote access… okay, my puns are not even remotely funny, but please keep reading to find out how I integrated my Zabbix with remote control.
Adding a new remote to Cozify
First things first! I found a spare smart remote control. These devices are handy — not based on infrared but operate on 433 MHz range, so there is no need to point exactly towards some device, instead, the device works everywhere in the Cozify hub range, which spans even to our backyard. In Cozify, I can create the actions that happen whenever I push a button: turn on/off one or multiple lights, change the active scene to something else, pretty much everything I can do in Cozify.
For a normal person, the functionality provided by Cozify would be more than sufficient. Me? I only use Cozify as a bridge between the remote control and Zabbix to do something Completely Else.
So, let’s fire up my iPhone Cozify app and add the remote first.
Connecting to Zabbix
Now that I have my remote added to Cozify, the next thing is to add it to Zabbix. The way I have implemented this is that a set of Python scripts is polling Cozify, and one of the scripts is polling for devices that have REMOTE_CONTROL capability. Of course, my newly added Zabbix remote has that capability, and sure enough, here it is:
In other words, I pressed button number two when the Unix time was 1652242931624. That’s nice to know, but what to do with this info?
20+ years ago… wait, what, 20+ years ago really… yes, over twenty years ago Bomfunk MC’s released their Freestyler song and its music video where a guy controls his surroundings with remote control. Well, he uses an MP3 player for control, but still, that’s a remote control for sure.
Take a nostalgia trip down the lane and marvel at the beauty that is some Finnish subway stations and then get back to our scheduled program.
My newfangled Zabbix remote is now just a dummy device for Cozify and for Zabbix it does not carry the Freestyler powers yet, but let’s see what happens when we apply some additional love.
Adding additional value
Thanks to the power of Zabbix value mapping I can transform the button numbers into something more meaningful. Let’s first add some values on the Value mapping tab:
See? That’s totally useful. Imagine I would be cooking dinner and my wife would be walking the dog / doing some gardening and I would like to let her know the dinner is ready: sending her a message over Signal or walking to our backyard would be soooooo much work if I can just press #1 on my remote instead. How lazy can you go? Very, although I suspect I am not going to use this in real life. But you never know!
To use these value mappings, I next applied the new value mapping to Zabbix remote item:
Suddenly, the latest data makes much more sense:
It deserves its own dashboard
So, at this point, we have the data collected and transformed into a human-readable form. Great! To make this all more consumable, there’s still some more work to do.
First, I created a simple Grafana dashboard with only a single panel in it:
Ain’t that beautiful?
It also deserves its own alerting
It would not be perfect without alerting, so let’s create a trigger!
… and an action…
… and behold, Zabbix can inform about the button presses to any configured media, in this case, e-mail. Of course, instead/in addition to messaging, a trigger could as well run any script, which means endless possibilities.
Cozify devs, yo!
Currently, there’s an ugly delay before Zabbix reacts due to the fact that my scripts are polling Cozify. It would be fantastic if Cozify could nudge my Zabbix via SNMP traps, or if I even could configure a central log server in Cozify’s setting to make it send its logs to my Zabbix server in real-time; that would turn my home monitoring into a real-time thing instead of polling happening every five minutes, as Zabbix could just follow the log and react to events accordingly.
Would you like fries with that?
Sure, this is just dumb play. But let’s stop for a minute and think about actual real-world usage scenarios where it might actually be useful to combine physical buttons with Zabbix, albeit I admit even these use cases might be weird:
The Refill button in a pub table would alert the waiter that table X needs more beer
In a factory/warehouse, physical buttons could be used to mark problematic areas during a physical inspection. Is any engine not running? A gauge showing funny values? Press a button near it, Zabbix gets an alert, and an engineer can visit the site
Use it as a stopwatch to measure how long you have been working; one button would mean “Start working”, another “Stop working”
Cozify supports more than just remote controls: battery-powered smart wall switches you can install without being an electrician, smart keyfobs, and smart dimmers, so there would be definitely more areas to explore in this physical interaction space. And, I talk about Cozify all the time as that’s what I have, but I’m sure your similar smart hub could do the same as well.
Anyway, I’ve just demonstrated to you how Zabbix could be used for limited instant messaging or customer service. That’s some serious flexibility.
I have been working at Forcepoint since 2014 and never get tired of inventing new ways to communicate with Zabbix. — Janne Pikkarainen
Can you monitor a Philips OneBlade shaver with Zabbix? Of course, you can! But why do that and how to monitor a dumb device with zero IoT capabilities?
Welcome to my weekly blog: I get my bread and butter by being a monitoring tech lead in a global cyber security company, but I monitor my home for fun with Zabbix & Grafana and do some weird experiments.
We all know how the battery-operated shavers, toothbrushes and similar devices sound very energetic and trustworthy immediately after you have charged their battery to full. Over time (over not so long time) they start to sound tired, but technically you can still use them. Or, you think you can still use them, but instead, they will betray you and die in the middle of the operation. Zabbix to the rescue!
Sing to me, bad boy
To get an idea about the battery runtime left, I needed to somehow capture the sound frequency and analyze it. The recording part was easy — after I had charged my razor to full level, I did leave it running and recorded the sound with my iPhone Voice Memos.
But how to get the sound frequency? This is the part where the audio engineers of the world can laugh at me in unison.
At first, I tried with Audacity as traditionally it has done all the tricks I possibly need to do with audio. Unfortunately, I could not find a way to accomplish my dream with it, and even if I would have, I fear I would have to manually do something with it, instead of the automated fashion I’m wishing for.
I could see all kinds of frequencies with Audacity, but was not able to isolate the humming sound of Philips OneBlade, at least not to a format I could use with Zabbix. Yes, Audacity has macros and some functionality remotely from the command line, but I interrupted my attempts with it. If you can do stuff like this with Audacity, drop me a note, I’m definitely interested!
Here come the numbers
Then, after a bit of searching, I found out aubiopitch. It analyzes the sample and returns a proper heckton of numbers back to you.
Those are not GPS coordinates or lottery numbers. That’s a timestamp in seconds and the sound frequency in Hz. And, just by peeking at the file manually, I found out that the values around 100, plus-minus something, were constantly present in the file. Yes, my brains have developed a very good pattern matching algorithm when it comes to log files, as that’s what I have been staring at for the last 20+ years.
As my 30+ minutes sample contained over 300,000 lines of these numbers, I did not want to bother my poor little home Zabbix with this kind of data volume for my initial analysis. I hate spreadsheet programs, especially with data that spans to hundreds of thousands of rows or more, so how to analyze my data? I possibly could have utilized Grafana’s CSV plugin, but to make things more interesting (for me, anyway), I called to my old friend gnuplot instead. Well, a friend in a sense that I know that it exists and that I occasionally used it two decades ago for simple plotting.
There it is, my big long needle in a haystack! Among some other environmental sounds, aubiopitch did recognize the Philips soundtrack as well! What if I filter out those higher frequencies? Or at least attempt to, my gnuplot-fu is not strong.
Yes, there it is, the upper line steadily coming down. After my first recording, it looks like that with a full battery the captured frequency starts from about 115 Hz, and everything goes well until about 93 Hz, but if I would start to shave around that time, I would better be quick, as I would only have two to three minutes left before the frequency quickly spirals down.
This thing is not in “production” yet, because
I need to do more recordings to see if I get similar frequencies each time
I need to fiddle with iPhone Shortcuts to make this as automated as possible.
Anyway, I did start building a preliminary Zabbix template with some macros already filled in…
… and I have a connection established between my dear Siri and Zabbix, too; this will be a topic for another blog entry in the future.
I am hoping that I could get Siri to upload the Voice Memo automatically to my Zabbix Raspberry Pi, which then would immediately analyze the data with aubiopitch maybe with a simple incron hook, and Zabbix would parse the values. That part is yet to be implemented, but I am getting there. It’s just numbers, and in the end, I will just point Zabbix to a simple text file to gather its numbers or make zabbix_sender to send in the values. Been there, done that.
I have been working at Forcepoint since 2014 and for this post to happen I needed to use some razor-sharp thinking. — Janne Pikkarainen
By day, I am a monitoring technical lead in a global cyber security company. By night, I monitor my home with Zabbix and Grafana in very creative ways. But what has Zabbix to do with Blender 3D software or virtual reality? Read on.
Full-stack monitoring is an old concept — in the IT world, it means your service is monitored all the way from physical level (data center environmental status like temperature or smoke detection, power, network connectivity, hardware status…) to operating system status, to your application status, enriched with all kinds of data such as application logs or end-to-end testing performance. Zabbix has very mature support for that, but how about… full house monitoring in 3D and, possibly, in virtual reality?
Slow down, what are you talking about?
The catacombs of my heart do have a place for 3D modeling. I am not a talented 3D artist, not by a long shot, but I have flirted with 3D apps since Amiga 500 and it’s Real 3D 1.4, then later with Amiga 1200 a legally purchased Tornado 3D, and not so legally downloaded Lightwave. With Linux, so after 1999 for me, I have used POV-Ray about 20 years ago, and as Blender went open source a long time ago, I have tried it out every now and then.
So, in theory, I can do 3D. In practice, it’s the “Hmm, I wonder what happens if I press this button” approach I use.
Not so slow, get to the point, please
Okay. There are several reasons why I am doing this whole home monitoring thing.
I have been doing IT monitoring for 20+ years, so really, there is not much new for me. Don’t get that wrong — boring is GOOD when it comes to business monitoring. Your business does count on it, and it’s perfect that whatever you need to monitor, you can do it reliably and easily. But for me, it does not challenge my brain or get my creative juices flowing. Monitoring the 3D world sure does.
With my home Zabbix & Grafana, I can get as wild and childish as I ever want. Of course, not so much at work. (Though I admit that at work I did set up an easter egg Grafana dashboard called OnlyFans — it is literally showing how the cooling fans of our servers and other devices are doing).
I want to give y’all new ideas and motivation to take your monitoring to the next level.
I want to help raise Zabbix as a product to a whole new level from traditional IT monitoring to monitoring the environment we all live in — anyway, the future of monitoring will more and more be in the real world, too
2D or not 2D, that is the question
For traditional IT monitoring, 2D interface and 2D alerts are OK, maybe apart from physical rack location visualization, where it definitely helps if a sysadmin can locate a malfunctioning server easily from a picture.
For the Real World monitoring, it is a different story. I’m sure an electrician would appreciate if the alert would contain pictures or animations visualizing the exact location of whatever was broken. The same for plumbers, guards, whoever needs to get to fix something in huge buildings, fast.
Let’s get to it
Now that you know my motivation, let’s finally get started!
In my case, leaping Zabbix from 2D to 3D meant just a bunch of easy steps:
Model my home in Sweet Home 3D; it’s very easy to use and definitely easier for my back than my wife requesting “could we try out how the sofa would look like over there…?”
In Blender, relabel the interesting objects to match with the names in Zabbix
Hook Zabbix and Blender together with Python and Blender Python API, so Zabbix can change the alerting object somehow for its properties — change material, change color, add a glow effect, make it fire/smoke/explode, whatever
Ask Blender Python API to export the rendered results as PNG images and as X3D files
Home sweet home
Sweet Home 3D is a relatively easy-to-use home modeling application. It’s free, and already contains a generous bunch of furniture, and with a small sum, you’ll get access to many, many more items.
After a few moments, I had my home modeled in Sweet Home 3D.
Next, I exported the file to .obj format, recognized by Blender.
Will it blend?
In Blender, I created a new scene, removed the meme-worthy default cube, and imported the Sweet Home 3D model to Blender.
Oh wow, it worked! Next, I needed to label the interesting items, such as our living room TV to match the names in Zabbix.
You modeled your home. Great! But does this Zabbix —> Blender integration work?
Yes, it does. Here is my first “let’s throw in some random objects into a Blender scene and try to manipulate it from Zabbix” attempt before any Sweet Home 3D business.
Fancy? No. Meaningful? Yes. There’s a lot going on in here.
Through Python, Zabbix was able to modify a Blender scene and change some colors to red.
Blender rendered the scene in its headless server mode (without GUI), and saved the resulting PNG still frame.
The script ran by Zabbix did copy the image to be available for Zabbix UI (in my case, I created /assets/3d/ directory which contains everything relevant to this experiment).
Zabbix URL widget is showing the image.
My Zabbix is now consulting Blender for every severity >=Average trigger, and I can also run the rendering manually any time I want.
First, here’s the manual refresh.
Next, here is the trigger:
Static image result
Here is a static PNG image rendering result by Blender Eevee rendering engine. Like gaming engines, Eevee cuts some corners when it comes to accuracy, but with a powerful GPU it can do wonders in real-time or at least in near-real-time.
The “I am not a 3D artist” part will hit you now hard. Cover your eyes, this will hurt. Here’s the Eevee rendering result.
That green color? No, our home is not like that. I just tried to make this thing look more futuristic, perhaps Matrix-like… but now it looks like… well… like I would have used a 3D program. The red Rudolph the Rednose Reindeer nose-like thing? I imagined it would be a neatly glowing red sphere along with the TV glowing, indicating an alert with our TV. Fail for the visual part, but at least the alert logic works! And don’t ask why the TV looks so strange.
But you get the point. Imagine if a warehouse/factory/whatever monitoring center would see something like this in their alerts. No more cryptic “Power socket S1F1A255DU not working” alerts, instead, the alert would pinpoint the alert in a visual way.
There was supposed to be an earth-shattering VR! Where’s the VR?
Mark Zuckerberg, be very afraid with your Metaverse, as Zabbixverse will rule the world. Among many other formats, Blender can export its scenes to X3D format. It’s one of the virtual world formats our web browsers do support, and dead simple to embed inside Zabbix/Grafana. Blender would support WebGL, too, but getting X3D to run only needed the use of <x3d> tag, so for my experiment, it was super easy.
The video looks crappy because I have not done any texture/light work yet, but the concept works! In the video, it is me controlling the movement.
In my understanding, X3D/WebGL supports VR headsets, too, so in theory you could be observing the status of whatever physical facility you monitor through your VR headset.
Of course, this works in Grafana, too.
How much does this cost to implement?
It’s free! I mean, Zabbix is free, Python is free and Blender is free, and open source. If you have some 3D blueprints of your facility in a format Blender can support — it supports plenty — you’re all set! Have an engineer or two or ten for doing the 3D scene labeling work, and pretty soon you will see you are doing your monitoring in 3D world.
What are the limitations?
The new/resolved alerts are not updated to the scene in real-time. For PNG files that does not matter much, as those are static and Zabbix can update those as often as needed, but for the interactive X3D files it’s a shame that for now the scene will only be updated whenever you refresh the page, or Zabbix does it for you. I need to learn if I can update X3D properties in real-time instead of a forced page load.
Coming up next week: monitoring Philips OneBlade
Next week I will show you how I monitor a Philips OneBlade shaver for its estimated runtime left. The device does not have any IoT functionality, so how do I monitor it? Tune in to this blog next week at the same Zabbix time.
I have been working at Forcepoint since 2014 and never get bored of inventing new ways to visualize data.
By day, I monitor a global cyber security company for a living. By night, I monitor my home with Zabbix and Grafana. In this weekly blog series, I’m sharing my weird experiments and new ideas on how to utilize monitoring.
On Easter, we were not at home but doing Easter stuff, that week I did not implement any major functionality to my home monitoring environment. But while I’m brewing new weird features, here are some bits and pieces of what I have learned about my home, and not shown here earlier.
I’m watching you, TV
We have one of those ‘smart’ TVs, just like about every recent TV happens to be. The one we have is a Samsung 2021 model. And, of course, I monitor it.
On the last two-day graph above, value 1.0 means that our TV is awake and responding to ICMP ping. During the annotated short spikes the TV does not have its screen on, but it is just silently awake and doing something with the network — may be checking for firmware updates or sending telemetry?
Anyway, it is definitely doing that many times per day. I will need to snoop more closely on what the heck it is doing.
A longer period of responding to ping indicates that we are actually watching the TV (or me playing PS5).
Garage, or not to garage?
That time, when I was writing this blog post, the spring has finally come, so we were doing some spring cleaning at home; no need for heavy winter jackets to be in our hallway closet anymore and so forth. For some items, my wife wondered what would be the humidity percentage in our garage.
Zabbix & Grafana to the rescue! The graph below shows the humidity levels of our living room and garage.
So, our garage definitely is a more humid place, and for now, some humidity-sensitive items were left inside our house instead of the garage.
Don’t get lost, get a map
This part is very much of a work in progress and is lacking the majority of the IoT devices we have, but I am also building a visual network map of my home environment. The map below uses the traditional Zabbix network map, but if I manage to pull a rabbit or two out of my hat, during the upcoming weeks you will see something Completely Else. Stay tuned!
Next week I will show you a definitely very weird target to monitor if I just manage to figure out how to do it.
There’s an app for that
But what if I am not at home? Sure, for any serious situations like a freezer temperature rapidly rising my Zabbix will e-mail me, but what if I just want to browse around? Using the web interface via iPhone could be done but is definitely not very convenient, so I am using ZBX Viewer app for iPhone instead. It’s handy, it’s free and it works.
I have been working at Forcepoint since 2014 and never get bored of staying up to date about the status of my house. — Janne Pikkarainen
* Please note that this blog post was originally written in April and some events mentioned do not correspond to the actual date at the time of publication.
Can you monitor the regular use of facial cream with Zabbix? Of course, you can! Here’s how. This same method could be very useful for monitoring if the elderly remember to take their meds or so.
What the heck?
A little background story. My forehead has a tendency for dry skin, so I should be using facial cream daily. Of course, as a man, I can guarantee you that 100% of the days I remember to use the cream, I apply it, so in practice, this means about 40-50% hit ratio.
As lately I have been adding more monitored targets to my home Zabbix, one night my wife probably thought she was being snarky or funny when she said “One monitor I could happily receive data about would be how often you remember to use your facial cream.”
A monitoring nerd does not take such ideas lightly.
Howdy door sensor, would you like to do some work?
I found a spare magnetic door sensor and a handy box where to store the cream.
You can see where this is going. This totally beautiful prototype of my Facial Cream Smart Storage Box is now deployed to test. If I open or close the box, the door sensor status changes, thus the facial cream mercy countdown timer resets.
How does it work? And does it really work?
Cozify smart IoT hub is keeping an eye on the magnetic door sensor’s last status change. And look, that awesome brown tape does not bother the magnets at all, Cozify reported the status as changed.
Now that I got the Cozify part working, my Zabbix can then receive the last change time as in Unix time.
On my Grafana, there’s now this absolutely gorgeous new panel, converting the Unix time to the “How long ago the last event happened?” indicator.
So the dashboard part is now working. But that is not all we need to do.
Alerting and escalation
Dashboards and monitoring are not useful at all if proper alerts are not being sent out. I now have this new alert trigger action rule in place.
In other words, if I forget to apply the facial cream, I have a one-hour time window to apply it, or otherwise, the alert gets escalated to my wife.
Will this method work? Is my prototype box reliable? I will tell you next time.
I have been working at Forcepoint since 2014 and never get tired of finding out new areas to monitor. — Janne Pikkarainen
Welcome to my weekly blog about how I monitor my home with Zabbix. Like Batman, I have a casual day job as a monitoring tech lead, and by night I tinker around with my home Zabbix. (Except that Batman does not do monitoring, or who knows.)
Anyone using Zabbix knows how it can gather data from just about anywhere, and it can send its alerts to just about anything — pager systems like PagerDuty or OpsGenie, ticketing systems like Jira, e-mail, SMS, etc. Integrating with those takes minutes, is officially supported, very well documented, and would not make sense at home. But, what if at home I would like to show any possible alerts in a completely different way?
Zabbix, meet Star Wars
So, if I want to integrate my Zabbix with a screen saver, how would that work? And how long would that take? Is it even possible?
My friend, in the case of the good old xscreensaver you have many, many options. For its text-based screen saver modules, you can feed it a text file, or an URL from where it downloads the text to show on the screen. Making xscreensaver contents dynamic is easy.
For Zabbix, to make it send its alerts as text, you have many options. At least:
Configure an action that runs whatever command to save the alert to a text file; even echo would do
Let your Other System fetch the alerts over Zabbix API
Let your Other System fetch the alerts directly from the Zabbix database
Send out your alerts as e-mails and let your Other System parse those e-mails
Configure a new custom media type to do something
Use Zabbix real-time export functionality
Internals of my xscreensaver showcase
For this exercise, I decided to use the sixth option: Zabbix real-time export functionality.
What’s that, you ask?
It makes Zabbix save history, trends and/or triggers to JSON files, which any 3rd party program can then parse and utilize. Enabling it happens in practice by commenting out three lines in the Zabbix server config file, altering the path where you want the JSON files to be created, deciding the maximum size of the created files, and what kind of events you want to export. Restart the Zabbix server process, done.
My Zabbix is now running on Raspberry Pi 4. Then, I have a FreeBSD laptop for anything nerdy I want to do, and the FreeBSD laptop has xscreensaver for this demonstration.
So, my FreeBSD laptop does rsync the JSON files from Zabbix server every minute, extracts the event host name and trigger name using jq, and saves the output to text file. Surround that with header and footer text files, and you are done.
Now every time I don’t touch my FreeBSD laptop in a while, it turns on its screensaver and shows me the recent Zabbix alerts. Zabbix, meet Star Wars.
For now, the alert format shown on scroller is not perfect, but it works and took three or four lines of bash in total to accomplish. That’s easy, and in total took maybe 15 minutes to implement to its current stage.
Other news about my home monitoring project
My facial cream usage monitoring is going great! I have not missed applying my facial cream even once — I mean, I have received alerts from Zabbix, but each and every time I have then proceeded to apply the cream and my wife has received zero alerts so far. Good boy, me! (Read more about this project next week!)
As the trains stopping at our station are not always reliable (they can be either very late or canceled), I now have a live map showing the real-time status of the trains we are interested in. This part actually does not have anything to do with Zabbix at the moment, it’s Grafana and its GraphQL plugin querying data from an official train traffic open data system.
Zabbix 6.0 gained a new official weather template, so I now have a local weather dashboard as well provided by Zabbix.
I made a “home status shown as emojis” dashboard to make monitoring fun(?) and interesting(?) for the whole family — now our home status can be observed from our living room TV easily.
Some screenshots are below:
I have been working at Forcepoint since 2014 and never get bored of showing the alerts in new ways. — Janne Pikkarainen
This week, I advanced my project on multiple fronts, so welcome to this little smorgasbord of different topics. In my future posts, I will go deeper into each topic as my project goes forward.
Zabbix the weatherman
Let me begin with a monitoring blooper.
As Zabbix has very well-working forecast/prediction functions for your usual IT capacity trending, I tried what happens if I let it predict the outdoor temperature based on recent temperatures. On my first try, this did not go as I planned.
You see, currently, here in Finland the temperatures change a lot during a 24 hours period: from nightly -10C or below temperatures to maybe +5C to +10C during the day. As I asked Zabbix to predict the weather based only on one hour of data one day ago, this did not go as planned.
OK, clearly the one hour worth of data was too little. What if ask Zabbix to base its forecast on one week worth of data?
The prediction slightly improves — at least it won’t predict a nuclear winter anymore — but only slightly. Zabbix in its little mind has no idea that the weather could get warmer due to the springtime. Or, in case Zabbix was right, I’ll let you know in a week.
Average data for Joe Average
As my monitoring setup collects more data, one thing I can get out of it will be averages. What’s the average temperature? What’s the average for this and that?
Above shows the average data for the last 24 hours, and on my Grafana dashboard the values change dynamically based on the time period I choose on it.
Who wouldn’t need home SLA reports?
Everybody knows how The Suits love their reports. I have this mental image where I think during their mornings they are like
[x] warm bread
[x] orange juice
[x] classical music
[x] latest reports
And oh dear, their morning is ruined if the [x] is missing from the last entry. Poor Suits.
Anyway, as the recent Zabbix 6.0 brought us revamped Business Services Monitoring, why not use it for home monitoring, too? This part includes very much work in progress, but I will show you the current results.
When I’m finished, each room will be configured as its own Business Service. For now, I only have entered the room names and some other stuff. There is only one room with some actual content, for now, and it’s our bedroom. What happens if I click on it?
I will get to see if the lights and temperature are OK, both from a technical standpoint and for their values. In case the status would not be OK, the root cause column would show me the reason why everything is not OK — though I would not need to click my way this far, the data would be shown on the previous page already.
As for SLAs (Service Level Agreement, for example, if you promise that your service will be available 99.9% of the time, it better be or your customer will be a sad panda and yell at you), those are also a work in progress. Zabbix can be let to generate daily/weekly/whatever SLA reports for any of the configured Business Services. I have yet to build them, but I have one for my home router already.
Come on, it’s sunny, let’s go out, Zabbix!
True story: this morning my wife asked that could I add pollen monitoring to Zabbix. My non-technical wife is getting excited about home monitoring, too! (I think she’s only pretending. Still AWESOME!)
I still need to add pollen monitoring — the data is available as open data — but I initialized The Great Outdoors Monitoring in two other areas.
Where’s my train?
Just before creating this post, I proved to myself that I can show live train data on Grafana. I sure got a screenful, as I have not played around with GraphQL too much, and for now, I got way more trains than I planned to get, and the data contains extra fields I need to filter out with Grafana’s Organise Fields. Still, connection established! Wooooo!
What’s for lunch?
Only added one lunch restaurant for now, but in theory, I will receive an alert whenever the restaurant posts its new weekly lunch menu. Zabbix is configured to be a good netizen though and it will only try to fetch the menu every one hour on Monday morning, no point to poll them all week, so let’s see how this will work.
That’s all for now. See you next week!
I have been working at Forcepoint since 2014 and I am a walking monitoring unit. — Janne Pikkarainen
* Please note, that this blog post was originally written a few months ago, in early Spring, and the temperature records do not correspond to the actual weather at the time of publication.
In my previous blog post, I wrote about how I monitor my home with Zabbix. This week, I am showing how I utilize Grafana to visualize the data collected by Zabbix and what are my plans to further improve all this.
What’s on TV, honey?
First of all, one of the reasons I am building my home Grafana dashboards is that they can look fantastic. Combine that with the fact that nowadays it is super easy to cast your screen to the living room TV — or even access Grafana by using TV’s built-in web browser –, and you have one heck of a situational awareness screen. Not that it would really be needed at home, but hey, a real-time dashboard easily beats your average soap opera. I am sure my wife would not appreciate the idea that we would stare at Grafana all night long, but that is a different story altogether. I digress.
The other reason why I am building all this? I have monitored all kinds of IT stuff since 2001, and have done some very creative gymnastics with Nagios and Zabbix, so now it’s time to try out monitoring The Real World. So far I have found out it is very similar to monitoring IT (duh).
Let’s dive into details
Above you can see a glimpse of my overall status Grafana dashboard. That’s actually all I have now, though it scrolls down for a page or two more.
The page provides me some really interesting information from battery levels to light status to firmware status of our devices. I will create some sub-dashboards and a Grafana playlist (slideshow), so our living room Mission Control TV can then show all the nuts and bolts of our home. Actually, we only have one TV and again, I am sure my wife would not appreciate The Grafana TV Show for too long, but one can dream.
Implemented so far:
Smart power outlet on/off status
Smart light bulbs on/off status
Info if our kitchen speaker is playing or not
Reachability status of different IoT devices we have around
Firmware status (is an upgrade needed or not) of our IoT devices
Amount of light (lux) status reported by Philips Hue motion sensors
Battery level monitoring of IoT devices; very good info to know especially about the smoke alarm device
Temperature monitoring in different rooms and outdoors
Humidity monitoring in different rooms and outdoors
Tons of details about our home Internet router; operational status of network ports, incoming/outgoing bandwidth, uplink status, errors, uptime, memory, CPU, disk and so on reported over SNMP
For now, for the panels I chose to show a single stat and would like to see the timeline history of the values, I can quickly click on Explore and see my data in a different way. Explore is a very powerful feature of Grafana, so if you are a Grafana user and have not yet realized its potential, try it out!
Still to come
This public blog about monitoring my home kind of forces me to progress with it. So, here’s what is still to come:
Create a sensible Zabbix template; I have made some progress on investigating the JSON provided by Cozify, so stay tuned!
Buy a Raspberry Pi (that rhymes, yo) and move this setup from two virtual machines running on my ages-old MacBook Pro Retina mid-2012 to it. And, I gotta say, for a ten-year-old machine this MacBook is still fantastic!
For a Finn, a catastrophic, show-stopping missing feature is that our sauna is not monitored. AIEEE! Need to fix that.
The spring is coming and so is the gardening time. Not that I would understand anything about it, but I’m sure that this is an area my wife would totally approve — I’ll buy some sensors so we get alerted if our flowers and other plants are threatened by excessive heat and dryness.
Buy some air quality sensors so I can track the air quality both indoors and outdoors.
Extend the monitoring to cover not only our home, but nearby services as well. I already have a Python script that can tell me if our local train is gonna be late or is canceled, but that was for different reasons a long time ago and not even used in Zabbix or Grafana. However, inserting that data into Zabbix is trivial, so I will add that.
Add upcoming/active weather alerts to Grafana
Grafana is perfectly capable to display for example the lunch menus of the nearby restaurants, so why not?
I have worked at Forcepoint since 2014 and never get bored of visualizing and analyzing data. — Janne Pikkarainen
For those of you who know me, this should not come as a surprise: I absolutely love Zabbix. It gives me the ultimate freedom to monitor whatever I need to monitor and is flexible enough to be able to monitor absolutely everything you can imagine. It’s free, it’s open-source, and scales to whatever needs you might have.
What does a monitoring nerd who is a technical lead for monitoring in a global cyber security company do during his downtime? That’s a silly question, mind you. Of course, he monitors his home with a home Zabbix instance.
Temperatures at our home, measured by Cozify, data collected by Zabbix
We have had a Cozify home automation system in our household since 2017. It is a nice central hub that supports IoT devices from a plethora of vendors and a vast selection of device categories, ranging from Philips Hue lights to motion sensors to cameras to fire alarm systems. You can then configure actions on some other device based on actions on one device: for example, turn on a light if a motion sensor detects movement.
Cozify is a very capable device, but where it definitely lacks is monitoring and analytics about what’s going on underneath.
As a monitoring addict, that is something I simply cannot stand.
Let’s build a bridge between Cozify and Zabbix
Someone has built an unofficial Python library for communicating with Cozify API. The library is a bit limited in functionality, the most limiting factor being that it only supports read-only operations. However, for my monitoring purposes, that does not matter, as I anyway need to read data.
For my initial testing purposes, I wrote a couple of small Python scripts to gather temperature and humidity data from our temperature sensors, and one script to monitor the general availability of the different IoT devices we have around. The scripts are run from cron every five minutes, and the results are written to text files that Zabbix reads. Zabbix has master items for temperature, humidity and reachability files, and using the dependent items, it can populate the data for all the 40+ data points I have now using just three polls.
Benefits of such project
Other than the cool geek factor, what’s the benefit of monitoring your home IoT hub? There’s plenty!
I get to learn all kinds of patterns about our home status: temperatures, reliability of individual devices, and the amount of time any device has been on/off
I get notified immediately if a critical device, like a smoke alarm, does not function properly
I get notified if the battery level on any battery-operated IoT device is getting low and can react before a device dies
I can follow how quickly the battery is draining on some device
Still for me to do
The current implementation is way too manual. It would be possible to utilize Zabbix low-level discovery to parse the JSON received from Cozify, but if I just dump everything from it, it contains all the possible device categories with different parameters: Philips Hue lights do report everything from their current brightness/color settings to if their firmware has been upgraded, and then the temperature or motion sensors do report back completely different set of data. That makes creating the monitored items automatically in a sane way a bit difficult.
So, I need to think a bit and figure out how to make my Cozify template more automatic.
I also need to set up a home Grafana instance speaking to Zabbix. Zabbix is excellent at collecting the monitoring data and sending out alerts, but Grafana is the perfect partner for Zabbix to do all the analytics and eye candy.
I have 20+ years of sysadmin/monitoring experience. Forcepoint has been my landing spot since 2014, and there I have been a monitoring technical lead since 2016. Everything Linux/FreeBSD, Zabbix, Grafana and open source in general is close to my heart. So close, in fact, that monitoring is also my hobby and I do weird experiments with Zabbix & Grafana at home. — Janne Pikkarainen
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.