Prometheus allows us to have alerts with “alertmanager”, which provides integrations with some services of communication. But from my point of view, Prometheus UI or Alertmanager lacks from accesibility. This is why I’m going to make a dashboard that matches with every every need: a full stack of alerts, only for one specific team or simply for each service. This will assure also the basics for scalability in monitoring. I’m managing both solutions for now, I’m going to analize a different method for both exporting and improving visualization of the alerts in just one dashboard. Considering that you already have some alerts configured in Prometheus and running alertmanager:
Prometheus allows us to export your alerts using the query “Alerts”, it’s just that simple.
With Grafana we have a variety of methods for configuring and show the alerts: one of them is to create a table, write into the label query the string “alerts” and then it will export all the alerts in only one table.
It looks pretty similar to the traditional monitoring services, for example Nagios, Zabbix or Sensu with Uchiwa.
The main goal is to have a dashboard with enough accessibility to let us play with colors, so that allows us to give a cognitive solution requiring to pay attention and not letting go the alerts on the monitor.
For example, you create a new dashboard:
Choose the option to create a panel of the type singlestat
Edit the panel: first choose your Prometheus instance as database and then you write the query. Repeat for each alert that you need to display in the dashboard.
You should have something like this:
This way you can have alerts for namespaces in the case you use k8s. For more information, you can read the site of Prometheus and Grafana.
Thanks so much to @anitalmada for the corrections in the post.