Monitoring OpenStack instances with Prometheus

This article describes how you can use Prometheus to automatically monitor OpenStack instances.

I won’t go into the details of the Prometheus architecture except that it is a pull-based model and scrapes the monitored targets. You can either define those targets statically (in the Prometheus configuration file) or dynamically using one of the many supported service discovery mechanisms. In particular, Prometheus is able to query the Nova API to list all your Nova instances (virtual machines) and with the help to the powerful relabel_configs parameter, you can filter, amend and transform this list into a set of scraped targets.

Lets have a quick walk-through.

First of all, I’ve booted 2 Ubuntu-based instances in my OpenStack project: prometheus-instance-1 and prometheus-instance-2. And I’ve added the following prometheus_io_scrape and prometheus_io_port metadata to both instances:

Instance metadata in the Horizon UI

On prometheus-instance-1, I’ve started one Prometheus server with this configuration:

global:
  scrape_interval:     15s
  evaluation_interval: 15s
rule_files: []
scrape_configs:
  # Scrape Prometheus itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  # Scrape OpenStack instances
  - job_name: 'openstack'
    openstack_sd_configs:
      - identity_endpoint: https://openstack.example.com:5000/v2.0
        username: simon
        project_name: prometheus-lab
        password: supersecret
        role: instance
    relabel_configs:
      # Keep only active instances
      - source_labels: [__meta_openstack_instance_status]
        action: keep
        regex: ACTIVE
      # Keep only instances which are flagged for scraping
      - source_labels: [__meta_openstack_tag_prometheus_io_scrape]
        action: keep
        regex: 'true'
      # Update the scraping port if required
      - source_labels:
        - __address__
        - __meta_openstack_tag_prometheus_io_port
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      # Replace the default instance by the OpenStack instance name
      - source_labels: [__meta_openstack_instance_name]
        target_label: instance

Then I’ve started the node_exporter service (with the default configuration) on my 2 instances:

Node exporter metrics

Et voila! Prometheus has automatically created the 2 node_exporter targets:

OpenStack targets in Prometheus

And the metrics are flowing:

Idle CPU metric

A few comments to wrap-up:

• If you need to scrape multiple exporters for the same Nova instance (MySQL for instance), you can add another OpenStack SD job to the Prometheus configuration and use different OpenStack metadata keys (prometheus_io_scrape_mysql=true and prometheus_io_mysql_port=9104 for instance) in the relabel_configs rules.
• In the demo, Prometheus is running within the same subnet as the monitored OpenStack instances. If you want to run Prometheus from the “outside”, the setup gets more complicated but PushProx can probably help.
• The OpenStack service discovery mechanism is also able to scrape Nova hypervisors. Personally I find it less useful since you usually have some configuration management tool which can inject those hypervisors as static targets in the Prometheus configuration file.

Update:

Since Prometheus v2.5.0, the OpenStack service discovery generates one target per network interface of Nova instance. This can be inconvenient when your instances have multiple interfaces and only one of them should be scraped.

For instance, you have 2 Neutron subnets (internal and mgmt). An instance may be connected to either one subnet or both, but only one network interface should be scraped (either internal or mgmt). With relabeling, we can leverage the __meta_openstack_address_pool meta-label and use Nova tags to filter out the unnecessary targets like this:

relabel_configs:
# Keep only instances which are flagged for scraping
- source_labels: [__meta_openstack_tag_prometheus_io_scrape]
  action: keep
  regex: 'true'
...
# Keep the target if the prometheus_io_address_pool tag is "mgmt" and it matches with the current subnet.
- source_labels:
  - __meta_openstack_address_pool
  -__meta_openstack_tag_prometheus_io_address_pool
  action: replace
  regex: "mgmt;mgmt"
  target_label: __tmp_keep_target
  replacement: "true"
# Keep target if the prometheus_io_address_pool tag is "internal" and it matches with the current subnet.
- source_labels:
  - __meta_openstack_address_pool
  -__meta_openstack_tag_prometheus_io_address_pool
  action: replace
  regex: "internal;internal"
  target_label: __tmp_keep_target
  replacement: "true"
# If you have more subnets, add more relabel configuration stanzas here.
# If the prometheus_io_address_pool tag is empty, use "mgmt" by default.
- source_labels:
  - __meta_openstack_address_pool
  - __meta_openstack_tag_prometheus_io_address_pool
  - __tmp_keep_target
  action: replace
  regex: "mgmt;;"
  target_label: __tmp_keep_target
  replacement: "true"
# keep only targets that matched any of the above.
- source_labels: [__tmp_keep_target]
  regex: true
  action: keep

Again this proves how powerful relabeling configuration can be!

Thanks to the great RobustPerception team for all the inspiration on their blog!