Monitoring

Execute Scripts on Zabbix Agent

Monitor Applications with Zabbix

www.zabbix.com

Zabbix is an open-source distributed monitoring software tool. It uses featured alarming and notification mechanisms to allow users to manage e-mail based alerts for various events. It also offers excellent reporting and data visualization features based on the data that multiple alerts produce. This makes Zabbix ideal for capacity planning and more.

In this article, I will dive into how we can run customized scripts on the Zabbix host with the help of Zabbix agent2. But first, let us install Zabbix components with Docker containers:

1) Install Zabbix with Docker Containers

Since the default bridge network that docker uses unless we specify another one while creating a container cannot provide automatic DNS resolution between containers and the -link flag is a legacy feature, we need to create an additional bridge network

docker network create --driver bridge zabbix-network

Now install MySQL container. Here mount a volume to prevent any data loss when the container is removed by any accident

docker run --network zabbix-network -dt --name mysql-server \
--mount type=volume,source=mysql,target=/var/lib/mysql \
-e MYSQL_DATABASE="zabbix" -e MYSQL_USER="zabbix" \
-e MYSQL_PASSWORD="evren123" \
-e MYSQL_ROOT_PASSWORD="YjA0OTYajskjadhBiN2EwNWFjMTRjZGU3Yjcy" \
--restart unless-stopped mysql --character-set-server=utf8 \
--collation-server=utf8_bin \
--default-authentication-plugin=mysql_native_password

Now start the Zabbix server instance. If you want to specify additional parameters, take a look at the environment variables section in https://hub.docker.com/r/zabbix/zabbix-server-mysql. For instance, based on the time your workloads take to run, you can set ZBX_TIMEOUT as an environment variable. By default, the value is 3 seconds. And also consider using ZBX_CACHESIZE for massive workloads

docker run --network zabbix-network --name zabbix-server-mysql \
-dt -e DB_SERVER_HOST="mysql-server" \
-e MYSQL_DATABASE="zabbix" \
-e MYSQL_USER="zabbix" -e MYSQL_PASSWORD="evren123" \
-e MYSQL_ROOT_PASSWORD="YjA0OTYajskjadhBiN2EwNWFjMTRjZGU3Yjcy" \
-e ZBX_TIMEOUT="10" -e ZBX_CACHESIZE=1G \
--hostname zabbix-server-mysql \
--restart unless-stopped zabbix/zabbix-server-mysql

Wait 30 seconds after installing zabbix-server-mysql and then install zabbix-web-nginx-mysql container to monitor the system

docker run --network zabbix-network \
--name zabbix-web-nginx-mysql -dt \
-e DB_SERVER_HOST="mysql-server" \
-e MYSQL_DATABASE="zabbix" \
-e MYSQL_USER="zabbix" -e MYSQL_PASSWORD="evren123" \
-e MYSQL_ROOT_PASSWORD="YjA0OTYajskjadhBiN2EwNWFjMTRjZGU3Yjcy" \
-e ZBX_SERVER_HOST="zabbix-server-mysql" \
--restart unless-stopped -p 8081:8080 \
zabbix/zabbix-web-nginx-mysql

Now install zabbix-agent2. Here also you can add additional parameters as environment variables, please check https://hub.docker.com/r/zabbix/zabbix-agent. For example, you can specify the timeout. Be sure to match it with the timeout of zabbix-server-mysql in order to avoid inconsistencies

docker run --network zabbix-network --name zabbix-agent \
-dt --hostname zabbix-agent -e ZBX_TIMEOUT="10" \
-e ZBX_HOSTNAME="Zabbix server" \
-e ZBX_ALLOWKEY="system.run[*]" \
-e ZBX_SERVER_HOST="zabbix-server-mysql" \
-e ZBX_SERVER_PORT="10051" --restart unless-stopped \
zabbix/zabbix-agent2

Access to Zabbix dashboard via http://<ip-of-your-host>:8081 and sign in with Username: Admin and Password: zabbix

For zabbix host to recognize zabbix-agent2, update the agent section like the one below

After that, wait 20–30 seconds and then the agent will be recognized

2) Run Scripts on Zabbix Agent2

When you put scripts into zabbix-agent2, do not forget to give them the necessary permissions. For instance either use “chmod +x <your-script>” or “chown zabbix <your-script>”

Go to Configuration, Hosts, and then Items

Click on “Create Item” and fill in the rest accordingly. Add a name, choose type as “Zabbix agent”, select “Key” as “system.run[command,<mode>]” and put the command inside to run your script, leave “Host interface” as “zabbix-agent:10050”, select the “Type of information”, update the “interval” based on your needs

You can click on “Test” and see the result of your script

3) Demo

Now we will check the status of pods in a Kubernetes cluster via check_kubernetes.sh script.

First, you need to install kubectl, jq, and git

docker exec -it --user root zabbix-agent bash
apk update && apk upgrade
apk add jq git
wget https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/amd64/kubectl
chmod +x kubectl
mv kubectl /usr/bin/

Now download the script and your kubeconfig file to zabbix-agent

git clone https://github.com/agapoff/check_kubernetes.git
chmod +x check_kubernetes/check_kubernetes.sh
mv check_kubernetes/check_kubernetes.sh /etc/zabbix
### Download your kubeconfig file ###
exit #From the zabbix-agent container

After that, go to Zabbix and create an item. For “Key” part you can use “system.run[ /etc/zabbix/check_kubernetes.sh -K <path_to_your_kubeconfig> -m pods -N <your_namespace> -w 3 ]”. This item will monitor the cluster and show the containers that restarted more than 3 times in <your_namespace>

RESOURCES:

GitHub - agapoff/check_kubernetes: Nagios/Icinga/Zabbix style plugin for checking Kubernetes

Zabbix Documentation 5.4

Docker Hub

Docker Hub

Legacy container links