Monitoring OpenStack instances with Prometheus

This article describes how you can use Prometheus to automatically monitor OpenStack instances.

I won’t go into the details of the Prometheus architecture except that it is a pull-based model and scrapes the monitored targets. You can either define those targets statically (in the Prometheus configuration file) or dynamically using one of the many supported service discovery mechanisms. In particular, Prometheus is able to query the Nova API to list all your Nova instances (virtual machines) and with the help to the powerful relabel_configs parameter, you can filter, amend and transform this list into a set of scraped targets.

Lets have a quick walk-through.

First of all, I’ve booted 2 Ubuntu-based instances in my OpenStack project: prometheus-instance-1 and prometheus-instance-2. And I’ve added the following prometheus_io_scrape and prometheus_io_port metadata to both instances:

Instance metadata in the Horizon UI

On prometheus-instance-1, I’ve started one Prometheus server with this configuration:

scrape_interval: 15s
evaluation_interval: 15s
rule_files: []
# Scrape Prometheus itself
  - job_name: 'prometheus'
- targets: ['localhost:9090']
# Scrape OpenStack instances
  - job_name: 'openstack'
- identity_endpoint:
username: simon
project_name: prometheus-lab
password: supersecret
role: instance
# Keep only active instances
      - source_labels: [__meta_openstack_instance_status]
action: keep
regex: ACTIVE
# Keep only instances which are flagged for scraping
      - source_labels: [__meta_openstack_tag_prometheus_io_scrape]
action: keep
regex: 'true'
# Update the scraping port if required
      - source_labels: [__address__, __meta_openstack_tag_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
# Replace the default instance by the OpenStack instance name
      - source_labels: [__meta_openstack_instance_name]
target_label: instance

Then I’ve started the node_exporter service (with the default configuration) on my 2 instances:

Node exporter metrics

Et voila! Prometheus has automatically created the 2 node_exporter targets:

OpenStack targets in Prometheus

And the metrics are flowing:

Idle CPU metric

A few comments to wrap-up:

  • If you need to scrape multiple exporters for the same Nova instance (MySQL for instance), you can add another OpenStack SD job to the Prometheus configuration and use different OpenStack metadata keys (prometheus_io_scrape_mysql=true and prometheus_io_mysql_port=9104 for instance) in the relabel_configs rules.
  • In the demo, Prometheus is running within the same subnet as the monitored OpenStack instances. If you want to run Prometheus from the “outside”, the setup gets more complicated but PushProx can probably help.
  • The OpenStack service discovery mechanism is also able to scrape Nova hypervisors. Personally I find it less useful since you usually have some configuration management tool which can inject those hypervisors as static targets in the Prometheus configuration file.

Thanks to the great RobustPerception team for all the inspiration on their blog!