Securing Prometheus Scrapes with the Kuma Service Mesh
Prometheus, the excellent open-source time series database and CNCF graduate, is the heart of many cloud-native monitoring stacks.
As a pull-based system, to get data in you must periodically scrape applications that expose metrics. These applications, or targets in Prometheus terms, expose metrics on HTTP endpoints. Easy enough, but how do we secure those HTTP requests?
Observability data is not always the first thought when securing systems but is crucially important — this data drives SLAs, business decisions, and key indicators to what applications are actually doing. We must be able to trust it!
We’ll look at two ways to ensure the security of your metrics: first with just the tools provided by Prometheus alone, then how the Kuma service mesh can complement and ease the burden.
Prometheus offers a highly-configurable HTTP client that supports a good set of options for authentication, like Basic Auth (username + password), OAuth, HTTP headers, and Mutual TLS (mTLS).
These cover a wide range of cases but have a couple drawbacks:
- Credentials, for all targets being scraped, must be accessible to the Prometheus server.
- Prometheus is usually scraping a wide swath of applications, so many different credentials must be centralized and continuously rotated.
This can quickly get unwieldy if you need to secure communication between Prometheus and applications split across teams and environments.
So how do we decentralize and ease the management of application credentials? If the network knew who was talking to who, and could enforce authorization rules between them, could we remove Prometheus’s knowledge of the credentials entirely?
With a service mesh like Kuma, a CNCF sandbox project, the network becomes a flexible, policy-driven engine that runs alongside applications. Kuma automatically instruments your services with everything needed to securely export their metrics. And with its attention to UX and community, you can get started within minutes and be supported throughout the process.
In the context of scraping, Kuma ensures that Prometheus has access only to exactly the applications we want it to. All we need is the mTLS and Traffic Permission policies and we’re off.
Prometheus 2.29 ships with a built-in kuma_sd that discovers all the applications inside a Kuma mesh, drastically simplifying how applications securely ship metrics. Configuration in Prometheus is as simple as:
kuma_sd_configs:
- server: http://kuma-control-plane.kuma-system.svc:5676With this, all your applications in Prometheus-enabled meshes will be securely scraped. And with mTLS and Traffic Permissions, you forget about the secrets :)
I’ve created a full example exists on Github, where you can see these policies in action and play with them yourself:
I am a maintainer of Kuma and an engineer over @VervericaData. I’m based in NYC, love food, and my dog Moody, and my wife Lily, order-independent. Find me on Twitter @austin_space_ce !
Thanks
This is the culmination of over a year of conversation and work between the Prometheus and Kuma communities.
Huge thank you to Julien Pivotto (@roidelaplue) for shepherding this contribution and his commitment to growing Prometheus!
And, of course, to Nikolay (@NTNikolaev), Jakub (@jdyszkiewicz), Ilya (@lobkovilya), and the rest of the Kuma team (@KumaMesh) 🎊
