Monitoring OpenStack instances with Prometheus
This article describes how you can use Prometheus to automatically monitor OpenStack instances.
I won’t go into the details of the Prometheus architecture except that it is a pull-based model and scrapes the monitored targets. You can either define those targets statically (in the Prometheus configuration file) or dynamically using one of the many supported service discovery mechanisms. In particular, Prometheus is able to query the Nova API to list all your Nova instances (virtual machines) and with the help to the powerful
relabel_configs parameter, you can filter, amend and transform this list into a set of scraped targets.
Lets have a quick walk-through.
First of all, I’ve booted 2 Ubuntu-based instances in my OpenStack project:
prometheus-instance-2. And I’ve added the following
prometheus_io_port metadata to both instances:
prometheus-instance-1, I’ve started one Prometheus server with this configuration:
# Scrape Prometheus itself
- job_name: 'prometheus'
- targets: ['localhost:9090']
# Scrape OpenStack instances
- job_name: 'openstack'
- identity_endpoint: https://openstack.example.com:5000/v2.0
# Keep only active instances
- source_labels: [__meta_openstack_instance_status]
# Keep only instances which are flagged for scraping
- source_labels: [__meta_openstack_tag_prometheus_io_scrape]
# Update the scraping port if required
- source_labels: [__address__, __meta_openstack_tag_prometheus_io_port]
# Replace the default instance by the OpenStack instance name
- source_labels: [__meta_openstack_instance_name]
Then I’ve started the
node_exporter service (with the default configuration) on my 2 instances:
Et voila! Prometheus has automatically created the 2
And the metrics are flowing:
A few comments to wrap-up:
- If you need to scrape multiple exporters for the same Nova instance (MySQL for instance), you can add another OpenStack SD job to the Prometheus configuration and use different OpenStack metadata keys (
prometheus_io_mysql_port=9104for instance) in the
- In the demo, Prometheus is running within the same subnet as the monitored OpenStack instances. If you want to run Prometheus from the “outside”, the setup gets more complicated but PushProx can probably help.
- The OpenStack service discovery mechanism is also able to scrape Nova hypervisors. Personally I find it less useful since you usually have some configuration management tool which can inject those hypervisors as static targets in the Prometheus configuration file.
Thanks to the great RobustPerception team for all the inspiration on their blog!