How to Create a Custom Warning in Zabbix

Zabbix actively monitors business infrastructure, but sometimes the built-in alerts do not cover all needs. In my case, I wanted to create a custom warning in Zabbix if a Btrfs snapshot hadn’t been completed in the last 24 hours on my Linux server. With this setup, I get an alert whenever my snapshot process fails or experiences a delay.

Why Use Custom Warnings in Zabbix?

Zabbix provides various alerts by default, but I needed a custom warning to:

• Ensure Btrfs snapshots are being created regularly
• Get notified if the snapshot process fails
• Integrate this check into my existing monitoring system

There are numerous various use-cases for custom warnings in Zabbix. Custom warnings provide greater flexibility, allowing you to adapt them to your specific monitoring needs.

Setting Up a Custom Warning in Zabbix

1. Create a Custom Script

First, I created a dedicated directory for custom scripts in Zabbix on my Linux host:

mkdir -p /etc/zabbix/scripts

Then, I created the script file and opened it for editing:

nano /etc/zabbix/scripts/check_btrfs_snapshot.sh

I added the following script to check if a Btrfs snapshot exists within the last 24 hours:

#!/bin/bash

SNAPSHOT_DIR="/mnt/btrfs/.snapshots"  # Adjust to your snapshot location
THRESHOLD=$(date -d '24 hours ago' +%s)

latest_snapshot=$(find "$SNAPSHOT_DIR" -maxdepth 1 -type d -printf '%T@ %p\n' | sort -nr | head -n1 | awk '{print $1}')

if [[ -z "$latest_snapshot" ]]; then
    echo 0  # No snapshots found
elif (( $(echo "$latest_snapshot >= $THRESHOLD" | bc -l) )); then
    echo 1  # Recent snapshot exists
else
    echo 0  # No recent snapshot
fi

After saving the script, I made it executable:

chmod +x /etc/zabbix/scripts/check_btrfs_snapshot.sh

2. Add the Script to Zabbix Agent 2

I’m using Zabbix Agent 2 for advanced features. Since Zabbix Agent 2 supports system.run, I modified zabbix_agent2.conf to add the script:

1. Open the configuration file: nano /etc/zabbix/zabbix_agent2.conf
2. Add this line to allow the script: AllowKey=system.run[/etc/zabbix/scripts/check_btrfs_snapshot.sh]
3. Restart the agent: systemctl restart zabbix-agent2

3. Create a Custom Item in Zabbix

1. Go to Zabbix Web Interface → Monitoring → Hosts
2. Select the host where Btrfs is monitored (or where you’re adding your custom script) and click Items.
3. Click Create Item and set your parameters:
   • Name: Btrfs Snapshot in the last 24 hours
   • Type: Zabbix Agent
   • Key: system.run[/etc/zabbix/scripts/check_btrfs_snapshot.sh]
   • Type of information: Text
   • Update interval: 30m

I also recommend testing your script to make sure it runs properly with the Test option at the bottom of the Item UI from above.

4. Set Up a Custom Trigger

1. Navigate to the Triggers tab.
2. Click Create Trigger.
3. Use this expression to trigger a warning if no snapshot exists in the last 24 hours:last(/example-hostname/system.run[/etc/zabbix/scripts/check_btrfs_snapshot.sh])=0.
   You can also use the “Add Condition UI” in the side panel by clicking Add to help generate the correct expression.
4. Set severity (e.g., Warning).
5. Click Add.

Final Thoughts

Setting up this custom warning in Zabbix ensures my Btrfs snapshots are created on schedule. If a snapshot fails, I get a warning immediately, allowing me to troubleshoot the issue before it becomes a bigger problem.

If you’re using Zabbix for monitoring, creating custom warnings like this can significantly improve your ability to track and respond to important system events. Try it out and enhance your monitoring experience!