Monitoring pending server updates with Zabbix
— by Andrea Grassi, Senior System Engineer at Caendra Inc.
Introduction
One of the many burdens of system administration is patching, as we need a way to monitor missing updates on our servers.
Currently we use the open source monitoring software Zabbix, which allows us to collect metrics automatically and create alerts based on them.
To do that, we created a script that checks the number of pending updates on a server (runs as a cron job) and saves the output to a file. Zabbix will then query this file.
There are a couple reasons we prefer this approach over invoking the script directly from Zabbix:
- We didn’t want to run the script many times a day
- We don’t want it to interfere with normal system administration locking the RPM/DEB directories too often
The script deployment, update and scheduling are automated on our infrastructure using Ansible so there’s no overhead on the management side. But that’s a story for another article 😃.
The script
Here we present the script which allows us to get the number of pending updates; we will break it in pieces to explain it more clearly.
#!/usr/bin/env bash
# From https://github.com/kvz/bash3boilerplate
# Require at least bash 3.x
if [[ "${BASH_VERSINFO[0]}" -lt "3" ]]; then echo "bash version < 3"; exit 1; fi# Exit on error. Append || true if you expect an error.
set -o errexit
set -o nounset# Bash will remember and return the highest exit code in a chain of pipes.
set -o pipefailPATH=/bin:/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
Up to this point, there’s nothing special, just some standard boilerplate code taken from bash3boilerplate. We encourage you to check it out and learn more about the project, as it’s very useful should you need a template for your bash scripts.
timestamp_file="/run/zabbix/caendra_check_update"
update_interval="86400" # 1 day
timestamp_file_mtime="0"
os=""
epoch=$(date "+%s")
tmpfile=$(mktemp --tmpdir=/run/zabbix)
outfile="/run/zabbix/zabbix.count.updates"function _detectOS {
if [[ -e /etc/centos-release ]]; then
export os="centos"
fi
if [[ -e /etc/debian_version ]]; then
export os="debian"
fi
}
The above snippet initializes some basic variables and defines a function to detect the running OS based on the existence of distro-specific files.
function _check_last_update {
if [[ ! -e $timestamp_file ]]; then
export update_needed=y
touch $timestamp_file
else
timestamp_file_mtime=$(stat -c %Y $timestamp_file )
fi if [[ "$((epoch-timestamp_file_mtime))" -gt "$update_interval" ]]; then
export update_needed=y
else
export update_needed=n
fi
}
This function limits our script to run the update operation at most once every 24 hours, to prevent conflicts with normal operations.
function _check_OS_upgrades {
if [[ "$os" == "debian" ]]; then
if [[ "$update_needed" == "y" ]]; then
apt update &>/dev/null
touch $timestamp_file
fi pkg_to_update=$((apt-get upgrade --simulate 2>&1 | wc -l) || true)
fiif [[ "$os" == "centos" ]]; then
if [[ ! -e /var/cache/yum/x86_64/7/base/repomd.xml ]]; then
# if the repomd.xml file does not exists,
# we assume that this is a new machine
# or "yum clean all" was run
export update_needed="y"
fi if [[ "$update_needed" == "y" ]]; then
# forced true as the --assumeno option
# always returns exit code 1
yum upgrade --assumeno &> /dev/null || true
touch $timestamp_file
fi yum_output=$(yum check-update --cacheonly && rc=$? || rc=$?; echo "rc=$rc" > $tmpfile)
source $tmpfile
rm $tmpfile if [[ "$rc" == "0" ]]; then
pkg_to_update="0"
fi if [[ "$rc" == "100" ]]; then
pkg_to_update=$(echo "$yum_output" | egrep -v '^(Load| \*|$)' | wc -l)
fi
fi
}
This function does the heavy lifting, based on the info gathered from the others functions it performs apt update (or yum update, accordingly to the detected distribution) and returns the number of packages which need upgrades.
_detectOS
_check_last_updatepkg_to_update=""_check_OS_upgradesecho "$pkg_to_update" > $outfile
For the last step, we gather all the needed information and put it into a file that Zabbix will read.
Now you can save the script in the /opt/zabbix_scripts directory. Make sure to check it out to ensure it’s working correctly before proceeding with the next step.
If you just want to download the full script, you can download it here from our code repository.
Script schedule
Once our script is ready and working correctly, we need it to run on a regular basis, so we’ll add a simple script to the /etc/cron.daily directory:
root@zabbix:~# cat /etc/cron.daily/zabbix_check_pending_updates
#!/bin/sh
bash "/opt/zabbix_scripts/check_updates.sh"This script must be marked as executable with the command:
chmod +x /etc/cron.daily/zabbix_check_pending_updatesZabbix client configuration
The last step is to add a UserParameter to the Zabbix agent configuration.
You must add the following single line to your /etc/zabbix/zabbix_agentd.conf. Alternatively you can create a new file under /etc/zabbix/zabbix_agentd.d with only this snippet.
UserParameter=os.updates.pending,cat "/run/zabbix/zabbix.count.updates"The above will configure Zabbix to execute our script to get the value of a new parameter named os.updates.pending.
Finally you can now check that everything works:
[root@zabbix:~] # zabbix_agentd -p | grep os.updates.pending os.updates.pending [t|4]If you get a result similar to the above, your configuration is ready, and you can now restart your Zabbix agent daemon.
You can of course add the metric to your templates, and create the various alarms/escalations accordingly. Here what the metric looks like when graphed out in Zabbix!
That’s it for now! We want to your from you so feel free to leave a feedback and let us know what you think about this approach.
