We have been using Prometheus as our monitoring solution for all of our Kubernetes clusters for a while. While providing us with much valuable insight into our services, it still retains some flaws. Prometheus lacks an option to override alert rules.
There’s practically no easy way to design alerting rules that can be altered or disabled for specific cases, for example when you need to disable certain alerts for particular deployments or change the thresholds for different namespaces, clusters, etc.
Let’s see how we were able to overcome this obstacle.
We primarily use Prometheus and Grafana as our monitoring solution…
Cloud DevOps engineer at LabyrinthLabs. Helping companies build, run, deploy and scale software and infrastructure by embracing the right technologies.