Prometheus & Microsoft Teams Integration

If you’re reading this article then you probably know that monitoring without alerting is useless, just as it’s useless to be alerted about everything.

Choosing the right alert and incident management tool is key to a successful monitoring experience. While commercial tools (Datadog, Pagerduty …) offer alerting functionality and integration with third-party platforms as out-of-the-box features, using open-source projects requires setting each component separately. At the end of this tutorial, you will know how to easily and efficiently set up the integration between Prometheus and Microsoft Teams.

Requirements:

• Kubernetes
• prometheus stack deployed to your cluster
• Helm

Designing your alert

Believe it or not, alerts should be designed, in a way to get the full benefit from theme, and get the Direct attention of those being notified.

To know more about designing alerts, I highly recommend you this talk from Monitorama conference

As you can see, this alert has all the mentioned aspects of a good alert, and Its resolution can be automated with the action “Unseal Vault”.

When an incident is resolved, people should know:

You can design your own card templates using MS Teams, or you can use this ones and adapt them for your needs.

Teams Configuration

From your notification channel in Teams

Manage connectors

Select the Incoming Webhook connector from the list of available connectors. Enter a name to identify this webhook later. You can also add an image which will be visible whenever a message is posted using this webhook. Click Create.

Add a webhook connector

Connector URL

Deploy to Kubernetes cluster

The prometheus stack uses AlertManager for managing alerts (grouping, silencing, sending …).
Alertmanager doesn’t support sending to Microsoft Teams out of the box. Fortunately, they allow you to use a generic webhook_config for cases like this. So this project fills in the missing part 🎉

We’ll use Helm for deployment:

$ helm repo add prometheus-msteams https://prometheus-msteams.github.io/prometheus-msteams/

Create your helm config file:

# config.yaml
---
replicaCount: 1
image:
  repository: quay.io/prometheusmsteams/prometheus-msteams
  tag: v1.5.1
connectors:
# in alertmanager, this will be used as http://prometheus-msteams:2000/prometheus-msteams. Replace with value you got from above steps
- sec-teams-alert: https://outlook.office.com/webhook/xxxx/xxxx
# extraEnvs is useful for adding extra environment variables such as proxy settings
extraEnvs:
  HTTP_PROXY: http://corporateproxy:8080
  HTTPS_PROXY: http://corporateproxy:8080

By default, the app uses a default card template to convert incoming Prometheus alerts to teams message cards. Whether you decide to create your own teams card template or you chose one from here, you’ll need to use customCardTemplate and set Its content with --set-file flag to set the value from the template file.

$ helm upgrade --install prometheus-msteams \
  --namespace monitoring -f config.yaml \
  --set-file customCardTemplate=custom-card.tmpl \
  prometheus-msteams/prometheus-msteams

Configure AlertManager

From your monitoring namespace, modify Alertmanager configuration, add a new receiver and rules to forward alerts to prometheus-msteams.

For example in this configuration, I’m forwarding all vault alerts to the security team channel, and when the alert is resolved, a new notification is sent

Encode file content :

$ cat alertmanager-conf.yaml | base64

And finally modify Alertmanager secret in your monitoring namespace with the new value.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-main
  namespace: monitoring
type: Opaque
data:
  alertmanager.yaml: BASE64 ENCODED VALUE FROM THE ABOVE STEP
EOF

That’s it, you’ve now configured Alertmanager to send alerts to teams using a custom card template. Happy monitoring 😄.

Final thoughts:

Monitoring is not enough, alerting is not enough, it is continuous work and continuous improvement to detect blind spots in your IT systems. Open source tools can be your starting point, but establishing a culture of monitoring and global adoption of monitoring by all your IT teams (Dev, Ops, Sec, business owners…), will at some point require moving to commercial solutions, due to their intuitiveness and ease of use and implementation.