Zabbix LLD using arbitrary JSON lists

We use Zabbix for much of our metric collection and monitoring at Dioss Smart Solutions. It has proven itself to be very reliable and flexible enough to let us implement every custom metric that we need.

One of the more powerful advanced features is LLD, or low-level discovery. This feature lets Zabbix dynamically discover what it should start monitoring, based on some information it gathers from a JSON document.

When we needed to monitor the age of some files in a specific folder of some of our Linux systems, we thought of something interesting to try with LLD. Let's say there's some files inside /var/lib/misc/ and we want to know when they were last modified.

/var/lib/misc/
├── status-foo.txt - last modified on 2022-03-17 at 12:49:28
└── status-bar.txt - last modified on 2022-03-15 at 15:50:18

Here’s why we need LLD to do this: we don't know what the names of the files are, and how many there are. This is something we would need to discover.

In the past we would have create our own script that generates the appropriate JSON document for LLD, but it seems now we don’t have to do that anymore. There's a new item key in the Zabbix Agent that was recently introduced in version 6.0, called vfs.dir.get. When you put in a directory path, it gives you back a JSON document with all sorts of metadata for all files in that directory.

Let’s create an item with key vfs.dir.get[/var/lib/misc].

Zabbix 1.png

This results in the following document:

[
    {
        "basename": "status-foo.txt",
        "pathname": "/var/lib/misc/status-foo.txt",
        "dirname": "/var/lib/misc",
        "type": "file",
        "user": "root",
        "group": "root",
        "permissions": "0644",
        "uid": 0,
        "gid": 0,
        "size": 0,
        "time": {
            "access": "2022-03-18T13:18:31+0100",
            "modify": "2022-03-18T13:18:31+0100",
            "change": "2022-03-21T19:08:03+0100"
        },
        "timestamp": {
            "access": 1647605911,
            "modify": 1647605911,
            "change": 1647886083
        }
    },
    {
        "basename": "status-bar.txt",
        "pathname": "/var/lib/misc/status-bar.txt",
        "dirname": "/var/lib/misc",
        "type": "file",
        "user": "root",
        "group": "root",
        "permissions": "0644",
        "uid": 0,
        "gid": 0,
        "size": 0,
        "time": {
            "access": "2022-03-21T08:32:17+0100",
            "modify": "2022-03-21T08:32:17+0100",
            "change": "2022-03-21T19:08:35+0100"
        },
        "timestamp": {
            "access": 1647847937,
            "modify": 1647847937,
            "change": 1647886115
        }
    }
]
      

The filenames and last modified timestamp are right there, and it's already structured as an array.

Let’s set up a Discovery rule and make it dependent on this item.

Zabbix 2.png


We're close, but the JSON document does not meet Zabbix’ requirements for LLD quite yet.

In the past, there was a requirement that a JSON document used for LLD had to have a parent “data” object. Luckily this was changed in Zabbix 4.2 and now any JSON array will be accepted.

What we will need to add though, are Zabbix macros. Zabbix expects to see variable names in a syntax they call macros. They take the form of {#VARIABLENAME}. In our JSON document, the variable name we would want to take to name each item in the list would be “basename".

What we could have done was to add a preprocessing step with a regular expression on the item with key vfs.dir.get[/var/lib/misc]. You can do it even easier and cleaner though: The discovery rule configuration page has a tab for LLD macros, where you can create mappings between Zabbix macro names and JSONPath expressions.

The expression $.basename will get you the basename value of each item in the JSON array. Mapping this to a macro makes it so that Zabbix can use it for LLD. You’re free to choose your macro name. We chose {#STATUSFILE}.

Zabbix 3.png


Let’s now create an item prototype. This describes how each discovered item should be made. You use the macro {#STATUSFILE} in the name and key fields, because they should be unique. Zabbix will then substitute the macro for the actual name of the thing it discovered.

It’s also of type dependent and uses the same vfs.dir.get[/var/lib/misc] master item as the discovery rule itself.

Zabbix 4.png

Now comes the hardest part: telling Zabbix which part of the JSON document contains the value it needs to monitor. In other words: we need to explain to Zabbix we want the modification timestamp for every {#STATUSFILE} and where it can find it inside the document.

To do this, head over to the preprocessing step and add a JSONPath rule. This is the expression that will get the timestamp:
$..[?(@.basename == "{#STATUSFILE}")].timestamp.modify

It looks pretty complex, so let’s break it down.

$ This is the root of the expression
.. Detach and go to the parent of this item (the top of the list)
[?()] Use a filter expression
@ In the current node
.basename == "{#STATUSFILE}" get the item who’s “basename” equals the contents of the “{#STATUSFILE}” macro. Zabbix will substitute the actual value for the macro prior to executing the JSONPath query.
.timestamp Get the child object called “timestamp”
.modify Get the child object called “modify”

The resulting value of this expression will be a number, but when it comes back from the JSONPath parser, it will be surrounded in brackets. e.g. [1647885836]. To strip off the brackets and get clean integer, add a left trim and a right trim preprocessing step.

Zabbix 5.png


Additionally, we want to create some triggers to warn us when these files are too old. The fuzzytime() trigger function is perfect for this, since this works with unix timestamps.

Zabbix 6.png Zabbix 7.png


Now, when the master item is retrieved from the agent, discovery will create the items and triggers based on the prototypes:

Zabbix 8.png Zabbix 9.png


And data for the discovered items will become visible in the latest data overview.

Zabbix 10.png

This was a lengthy explanation, but the resulting YAML template is only 65 lines!

zabbix_export:
  version: '6.0'
  date: '2022-03-21T18:33:27Z'
  groups:
    -
      uuid: 791fd98178aa4e1bac5b104070c46c12
      name: Templates/Dioss
  templates:
    -
      uuid: fb560347d50348a18092944ec6792cce
      template: 'Example template'
      name: 'Example template'
      groups:
        -
          name: Templates/Dioss
      items:
        -
          uuid: 8baca74a70834cce85dbc4c192c2953f
          name: 'Contents of folder /var/lib/misc'
          key: 'vfs.dir.get[/var/lib/misc]'
          delay: 5m
          trends: '0'
          value_type: TEXT
      discovery_rules:
        -
          uuid: 81545204a675452aa1e4274e6a9196ac
          name: 'Discovery of files in /var/lib/misc'
          type: DEPENDENT
          key: discovery.files.var.lib.misc
          delay: '0'
          item_prototypes:
            -
              uuid: 79a19a43d94d42d0abc79392737bf5e6
              name: 'Last modified timestamp of status file {#STATUSFILE}'
              type: DEPENDENT
              key: 'statusfile.modified.timestamp[{#STATUSFILE}]'
              delay: '0'
              preprocessing:
                -
                  type: JSONPATH
                  parameters:
                    - '$..[?(@.basename == "{#STATUSFILE}")].timestamp.modify'
                -
                  type: LTRIM
                  parameters:
                    - '['
                -
                  type: RTRIM
                  parameters:
                    - ']'
              master_item:
                key: 'vfs.dir.get[/var/lib/misc]'
              trigger_prototypes:
                -
                  uuid: 6570fb3185ab41e8962a0031b436588d
                  expression: 'fuzzytime(/Example template/statusfile.modified.timestamp[{#STATUSFILE}],2d)=0'
                  name: 'Status file {#STATUSFILE} is older than 2 days'
                  priority: WARNING
                  manual_close: 'YES'
          master_item:
            key: 'vfs.dir.get[/var/lib/misc]'
          lld_macro_paths:
            -
              lld_macro: '{#STATUSFILE}'
              path: $.basename


GO BACK TO BLOGS

Scroll down