Alerts history in Prometheus

Today I found that Prometheus has history of all alerts (including pending state), and it’s right there, inside Prometheus itself.

Up to this point we had used a rather complicated alerts logger, but, turned out, it’s completely redundant.

The query is utterly trivial:

ALERTS

This is an example of output:

ALERTS{
 alert_class=”updates”,
 alertname=”reboot required”,
 alertstate=”firing”,
 instance=”10.0.0.1:11111",
 job=”node_exporter”,
 severity=”info”
} 1

that’s all. Enjoy.