Zabbix LLD using arbitrary JSON lists

We use Zabbix for much of our metric collection and monitoring at Dioss Smart Solutions. It has proven itself to be very reliable and flexible enough to let us implement every custom metric that we need.

One of the more powerful advanced features is LLD, or low-level discovery. This feature lets Zabbix dynamically discover what it should start monitoring, based on some information it gathers from a JSON document.

When we needed to monitor the age of some files in a specific folder of some of our Linux systems, we thought of something interesting to try with LLD. Let's say there's some files inside /var/lib/misc/ and we want to know when they were last modified.

Here’s why we need LLD to do this: we don't know what the names of the files are, and how many there are. This is something we would need to discover.

In the past we would have create our own script that generates the appropriate JSON document for LLD, but it seems now we don’t have to do that anymore. There's a new item key in the Zabbix Agent that was recently introduced in version 6.0, called vfs.dir.get. When you put in a directory path, it gives you back a JSON document with all sorts of metadata for all files in that directory.

Let’s create an item with key vfs.dir.get[/var/lib/misc].

The filenames and last modified timestamp are right there, and it's already structured as an array.

Let’s set up a Discovery rule and make it dependent on this item.

We're close, but the JSON document does not meet Zabbix’ requirements for LLD quite yet.

In the past, there was a requirement that a JSON document used for LLD had to have a parent “data” object. Luckily this was changed in Zabbix 4.2 and now any JSON array will be accepted.

What we will need to add though, are Zabbix macros. Zabbix expects to see variable names in a syntax they call macros. They take the form of {#VARIABLENAME}. In our JSON document, the variable name we would want to take to name each item in the list would be “basename".

What we could have done was to add a preprocessing step with a regular expression on the item with key vfs.dir.get[/var/lib/misc]. You can do it even easier and cleaner though: The discovery rule configuration page has a tab for LLD macros, where you can create mappings between Zabbix macro names and JSONPath expressions.

The expression $.basename will get you the basename value of each item in the JSON array. Mapping this to a macro makes it so that Zabbix can use it for LLD. You’re free to choose your macro name. We chose {#STATUSFILE}.

Let’s now create an item prototype. This describes how each discovered item should be made. You use the macro {#STATUSFILE} in the name and key fields, because they should be unique. Zabbix will then substitute the macro for the actual name of the thing it discovered.

It’s also of type dependent and uses the same vfs.dir.get[/var/lib/misc] master item as the discovery rule itself.

Now comes the hardest part: telling Zabbix which part of the JSON document contains the value it needs to monitor. In other words: we need to explain to Zabbix we want the modification timestamp for every {#STATUSFILE} and where it can find it inside the document.

To do this, head over to the preprocessing step and add a JSONPath rule. This is the expression that will get the timestamp:
$..[?(@.basename == "{#STATUSFILE}")].timestamp.modify

The resulting value of this expression will be a number, but when it comes back from the JSONPath parser, it will be surrounded in brackets. e.g. [1647885836]. To strip off the brackets and get clean integer, add a left trim and a right trim preprocessing step.

Additionally, we want to create some triggers to warn us when these files are too old. The fuzzytime() trigger function is perfect for this, since this works with unix timestamps.

Now, when the master item is retrieved from the agent, discovery will create the items and triggers based on the prototypes:

And data for the discovered items will become visible in the latest data overview.