We have been using Prometheus as our monitoring solution for all of our Kubernetes clusters for a while. While providing us with much valuable insight into our services, it still retains some flaws. Prometheus lacks an option to override alert rules.

There’s practically no easy way to design alerting rules that can be altered or disabled for specific cases, for example when you need to disable certain alerts for particular deployments or change the thresholds for different namespaces, clusters, etc.

Let’s see how we were able to overcome this obstacle.

Our Monitoring Setup

We primarily use Prometheus and Grafana as our monitoring solution…

Martin Hauskrecht

Cloud DevOps engineer at LabyrinthLabs. Helping companies build, run, deploy and scale software and infrastructure by embracing the right technologies.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store