Image for post
Image for post

Alertmanager (Prometheus) notification configuration in Kubernetes

Prometheus is open-source monitoring system which since 2016 is a part of Cloud Native Computing Foundation. It became a second member of it after Kubernetes. That’s why Prometheus could be pretty easily setup and configured.

Usually Prometheus is used in cooperation with Alertmanager (for routing notification to different popular IM services, filtering of its etc) and Grafana (allows you to query, visualize metrics from Prometheus backend). And today I am gonna show you how to configure Alertmanager to send emails in case of troubles with your Kubernetes installation.

My cluster was setup with Kops in AWS and it’s really wonderful tool for creating HA Kubernetes in cloud. For Prometheus setup I’ve used next addon description https://github.com/kubernetes/kops/tree/master/addons/prometheus-operator. After deploying it creates monitoring namespace and next secrets:

We are interested only in alertmanager-main secret because there live Alertmanager settings. Let’s look at them:

alertmanager.yaml field is encoded with base64, let’s see what’s inside:

Obviously for now Alertmanager doesn’t send anything at all. Correct config needs to be looked as follow:

So we are going to send emails from outlook.com relay (from-email@outlook.com emailbox) with password y0ur-v3ry-h@rd-p@ssw0rd to to-email@example.com. But that’s not all. Except that we need to encode this config back and change alertmanager-main secret. For this we will create basic template:

And with next command we will replace ALERTMANAGER_CONFIG variable with value and at once upload new secret to K8s:

In Kubernetes (using kubectl) or Alertmanager we could see if changes were applied correctly:

Image for post
Image for post

Now supposing we have such alert:

Image for post
Image for post

And soon we get emails like this:

Image for post
Image for post

That’s it. Hope this small article was helpful for you. Enjoy your cluster!

Links (say thanks those guys too):

Written by

DevOps in a galaxy far, far away…

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store