Custom Notifications with Alert Manager’s Webhook Receiver in Kubernetes

Prometheus’s AlertManager receives the alerts send from Prometheus’ alerting rules, and then manages them accordingly. One of the action is to send out external notifications such as Email, SMS, Chats. Out of box, AlertManager provides a rich set of integrations for the notifications. However, in real life projects, these lists are always not enough. For my case, I need to send out email through Gmail and SMS through Twilo.

Quoted from AlertManager’s documentation

We’re not actively adding new receivers, we recommend implementing custom notification integrations via the webhook receiver

Fair enough, I will create my own webhook receiver for Gmail and Twilo.

The Kubernetes here used is the one with IBM Cloud Private V 2.1.

1. Configure Receiver in Prometheus

First let’s define the alertmanager’s configuration as a configmap in Kubernetes, and apply it through the kubectl command line.

apiVersion: v1
kind: ConfigMap
metadata:
name: monitoring-prometheus-alertmanager
namespace: kube-system
data:
alertmanager.yml: |-
global:
receivers:
- name: default-receiver
- name: ycap-webhook
webhook_configs:
- url: "http://am-webhook.ycap/webhook"
    route:
group_wait: 10s
group_interval: 5m
receiver: default-receiver
repeat_interval: 3h

routes:
- receiver: ycap-webhook
match_re:
app: ycap

Other than the default-receiver, I defined my ycap-webhook receiver, which is using webhook-configs and the URL is pointing to http://am-webhook.ycap/webhook The hostname am-webhook.ycap is the Kubernetes’ service name that I am going to create later together with the custom webhook Deployment in the namespace of ycap .

If the alert regex matches the label of app with the value of ycap , it will be routed to the ycap-webhook receiver to handle.

2. Alert Rule

Secondly I define the alert rules to generate alert. Create the following configmap and apply it with kubectl similarly.

apiVersion: v1
kind: ConfigMap
metadata:
name: monitoring-prometheus-alertrules
namespace: kube-system
data:
ycap_rules.yml: |-
groups:
- name: ycap
rules:
- alert: HighSystemLoad
expr: systemload_average > 90
for: 5s
labels:
app: ycap
severity: critical
annotations:
summary: "High system load: {{ $value | printf \"%.2f\" }}%"
emails: "user1@gmail.com,user2@xyz.com"
phones: "+6512345678, +6522345678"

Here I generate an alert called as “HighSystemLoad” if the following expression systemload_average > 10 evaluated as true.

I attached the labels with ‘ycap’ and assign the severity as ‘critical’. In the meantime, I created some annotations for the alert, such as the summary text, the list of notification recipients, to which the webhook receiver will send out notification.

3. Webhook Program

Thanks to Prometheus, the creation of webhook is straightforward. When the alert is send from AlertManager to the target webhook, the json data can be decoded using the go module of github.com/prometheus/alertmanager/template. You can refer to this godoc for more detail.

Following is the webhook golang http handler.

func webhook(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
  data := template.Data{}
if err := json.NewDecoder(r.Body).Decode(&data); err != nil {
asJson(w, http.StatusBadRequest, err.Error())
return
}
log.Printf("Alerts: GroupLabels=%v, CommonLabels=%v", data.GroupLabels, data.CommonLabels)
for _, alert := range data.Alerts {
log.Printf("Alert: status=%s,Labels=%v,Annotations=%v", alert.Status, alert.Labels, alert.Annotations)
  severity := alert.Labels["severity"]
switch strings.ToUpper(severity) {
case "CRITICAL":
gmailSend(alert)
sms(alert)
case "WARNING":
gmailSend(alert)
default:
log.Printf("no action on severity: %s", severity)
}
}
  asJson(w, http.StatusOK, "success")
}
func healthz(w http.ResponseWriter, r *http.Request) {
fmt.Fprint(w, "Ok!")
}
func main() {
http.HandleFunc("/healthz", healthz)
http.HandleFunc("/webhook", webhook)
 listenAddress := ":8080"
if os.Getenv("PORT") != "" {
listenAddress = ":" + os.Getenv("PORT")
}
 log.Printf("listening on: %v", listenAddress)
log.Fatal(http.ListenAndServe(listenAddress, nil))
}

Once the alert template data is obtained, we can process its labels and annotations. If the severity is CRITICAL, we send both email and SMS. The recipients are based on the annotations.

The email is using Gmail services and SMS are sent with Twilio. I have some Gmail helper function to get the gmail service. While the Twilio SMS is simply calling the Twillio API.

func gmailSend(alert template.Alert) {
log.Printf("sending gmail...")
  gmailService, err := getGmailService()
if err != nil {
log.Fatalf("Unable to get Gmail service: %v", err)
}
  var message gmail.Message
  e := email.NewEmail()
e.From = os.Getenv("GMAIL_FROM")
emails := alert.Annotations["emails"]
reg := regexp.MustCompile("\\s*,\\s*")
e.To = reg.Split(emails, -1)
e.Subject = "ICP Email Notification"
e.Text = []byte(alert.Annotations["summary"])
  rawText, err := e.Bytes()
if err != nil {
log.Printf("error to convert into bytes: %v", err)
return
}
message.Raw = base64.URLEncoding.EncodeToString(rawText)
  _, err = gmailService.Users.Messages.Send("me", &message).Do()
if err != nil {
log.Printf("Error sending gmail: %v", err)
}
}
func sms(alert template.Alert) {
log.Printf("sending sms through twilio...")
twilio := gotwilio.NewTwilioClient(os.Getenv("TWILIO_ACCOUNT"), os.Getenv("TWILIO_TOKEN"))
from := os.Getenv("TWILIO_FROM")
message := alert.Annotations["summary"] + " Status: " + alert.Status
  to := alert.Annotations["phones"]
reg := regexp.MustCompile("\\s*,\\s*")
for _, r := range reg.Split(to, -1) {
if strings.TrimSpace(r) != "" {
_, _, err := twilio.SendSMS(from, r, message, "", "")
if err != nil {
log.Printf("error sending SMS: %v", err)
}
} else {
log.Printf("ignore empty recipient")
}
}
}

The detail of the webhook can be refereed to this Github repo.

4. Webhook Kubernetes Deployment

I build a docker image and push it to the Docker hub with the following Dockerfile.

FROM alpine:3.7
RUN apk update && apk add ca-certificates && rm -rf /var/cache/apk/*
WORKDIR /amhook
ADD amWebhook .
ADD getToken .
CMD ["./amWebhook"]

With the image pushed, we can then create the Kubernetes’ deployment and service for the AlertManager to forward the alerts to the receiver.

apiVersion: apps/v1
kind: Deployment
metadata:
name: am-webhook
namespace: ycap
labels:
app: am-webhook
spec:
replicas: 2
selector:
matchLabels:
app: am-webhook
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 50%
maxSurge: 1
template:
metadata:
labels:
app: am-webhook
spec:
terminationGracePeriodSeconds: 30
# imagePullSecrets:
# - name: admin.regkey
containers:
- name: am-webhook
image: zhiminwen/am-webhook:1.0.0@sha256:b04b263555bda4efd650993891d1bfe02548e6d5ddb492c2da127e755d188d3b
imagePullPolicy: IfNotPresent

env:
- name: PORT
value: "80"
- name: TWILIO_ACCOUNT
value: YOUR_ACCOUNT
- name: TWILIO_TOKEN
value: YOUR_ACCOUNT
- name: TWILIO_FROM
value: "YouNumber"
- name: GMAIL_FROM
value: "YourEmailID@gmail.com"
        livenessProbe:
httpGet:
path: /healthz
port: 80
readinessProbe:
httpGet:
path: /healthz
port: 80

resources:
limits:
cpu: 10m
memory: 30Mi
requests:
cpu: 10m
memory: 30Mi
        volumeMounts:
- name: config-volume
mountPath: /amhook/config

volumes:
- name: config-volume
configMap:
name: am-webhook-config

The config-volume is the configmap that mapped to the following file.

config/client_secret.json
config/token.json

These files are used for getting the Gmail tokens with the OAuth 2.0 client ID. I will describe them briefly in the last section.

The service.yaml is shown as below. With the service defined, it is why in the first section I can point the webhook to http://am-webhook.cap/webhook.

apiVersion: v1
kind: Service
metadata:
name: am-webhook
labels:
name: am-webhook
namespace: ycap
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: am-webhook

5. Gmail Services

The Gmail is a little bit complex. Followings are the brief steps to setup the authentication for gmail services.

  1. In the developer console, create the OAuth 2.0 client ID.

Download it as the client_secret.json file

2. Run the getToken program.

It will first read the client_secret.json file you saved, and create a URL. Copy the URL, launch a brower, Paste it, Authorize the Gmail access, then you will be given a token as below,

Paste it back to the getToken program, then the token.json file will be created.

You can then client the config map file for the K8s as shown as below based on the client_secret.json and token.json file

Conclusion

Just like human being progresses after grasping the Fire, Prometheus and AlertManager Webhook gives us the endless possibility for the integrations. For an on-premise kubernetes cloud, imagine we can integrate further with Email gateway, SMS gateway, trouble ticketing system…