Alertmanager (Prometheus) notification configuration in Kubernetes
Prometheus is open-source monitoring system which since 2016 is a part of Cloud Native Computing Foundation. It became a second member of it after Kubernetes. That’s why Prometheus could be pretty easily setup and configured.
Usually Prometheus is used in cooperation with Alertmanager (for routing notification to different popular IM services, filtering of its etc) and Grafana (allows you to query, visualize metrics from Prometheus backend). And today I am gonna show you how to configure Alertmanager to send emails in case of troubles with your Kubernetes installation.
My cluster was setup with Kops in AWS and it’s really wonderful tool for creating HA Kubernetes in cloud. For Prometheus setup I’ve used next addon description https://github.com/kubernetes/kops/tree/master/addons/prometheus-operator. After deploying it creates monitoring namespace and next secrets:
We are interested only in alertmanager-main secret because there live Alertmanager settings. Let’s look at them:
alertmanager.yaml field is encoded with base64, let’s see what’s inside:
Obviously for now Alertmanager doesn’t send anything at all. Correct config needs to be looked as follow:
So we are going to send emails from outlook.com relay (from-email@outlook.com emailbox) with password y0ur-v3ry-h@rd-p@ssw0rd to to-email@example.com. But that’s not all. Except that we need to encode this config back and change alertmanager-main secret. For this we will create basic template:
And with next command we will replace ALERTMANAGER_CONFIG variable with value and at once upload new secret to K8s:
$ sed "s/ALERTMANAGER_CONFIG/$(cat alertmanager.yaml | base64 -w0)/g" alertmanager-secret-k8s.yaml | kubectl apply -f -In Kubernetes (using kubectl) or Alertmanager we could see if changes were applied correctly:
Now supposing we have such alert:
And soon we get emails like this:
That’s it. Hope this small article was helpful for you. Enjoy your cluster!
Links (say thanks those guys too):
