Monitoring OpenStack instances with Prometheus

Simon Pasquier
Oct 13, 2017 · 4 min read

This article describes how you can use Prometheus to automatically monitor OpenStack instances.

I won’t go into the details of the Prometheus architecture except that it is a pull-based model and scrapes the monitored targets. You can either define those targets statically (in the Prometheus configuration file) or dynamically using one of the many supported service discovery mechanisms. In particular, Prometheus is able to query the Nova API to list all your Nova instances (virtual machines) and with the help to the powerful relabel_configs parameter, you can filter, amend and transform this list into a set of scraped targets.

Lets have a quick walk-through.

First of all, I’ve booted 2 Ubuntu-based instances in my OpenStack project: prometheus-instance-1 and prometheus-instance-2. And I’ve added the following prometheus_io_scrape and prometheus_io_port metadata to both instances:

Instance metadata in the Horizon UI

On prometheus-instance-1, I’ve started one Prometheus server with this configuration:

global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files: []scrape_configs:
# Scrape Prometheus itself
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
# Scrape OpenStack instances
- job_name: 'openstack'
openstack_sd_configs:
- identity_endpoint: https://openstack.example.com:5000/v2.0
username: simon
project_name: prometheus-lab
password: supersecret
role: instance
relabel_configs:
# Keep only active instances
- source_labels: [__meta_openstack_instance_status]
action: keep
regex: ACTIVE
# Keep only instances which are flagged for scraping
- source_labels: [__meta_openstack_tag_prometheus_io_scrape]
action: keep
regex: 'true'
# Update the scraping port if required
- source_labels:
- __address__
- __meta_openstack_tag_prometheus_io_port
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
# Replace the default instance by the OpenStack instance name
- source_labels: [__meta_openstack_instance_name]
target_label: instance

Then I’ve started the node_exporter service (with the default configuration) on my 2 instances:

Node exporter metrics

Et voila! Prometheus has automatically created the 2 node_exporter targets:

OpenStack targets in Prometheus

And the metrics are flowing:

Idle CPU metric

A few comments to wrap-up:

  • If you need to scrape multiple exporters for the same Nova instance (MySQL for instance), you can add another OpenStack SD job to the Prometheus configuration and use different OpenStack metadata keys (prometheus_io_scrape_mysql=true and prometheus_io_mysql_port=9104 for instance) in the relabel_configs rules.
  • In the demo, Prometheus is running within the same subnet as the monitored OpenStack instances. If you want to run Prometheus from the “outside”, the setup gets more complicated but PushProx can probably help.
  • The OpenStack service discovery mechanism is also able to scrape Nova hypervisors. Personally I find it less useful since you usually have some configuration management tool which can inject those hypervisors as static targets in the Prometheus configuration file.

Update:

Since Prometheus v2.5.0, the OpenStack service discovery generates one target per network interface of Nova instance. This can be inconvenient when your instances have multiple interfaces and only one of them should be scraped.

For instance, you have 2 Neutron subnets (internal and mgmt). An instance may be connected to either one subnet or both, but only one network interface should be scraped (either internal or mgmt). With relabeling, we can leverage the __meta_openstack_address_pool meta-label and use Nova tags to filter out the unnecessary targets like this:

relabel_configs:
# Keep only instances which are flagged for scraping
- source_labels: [__meta_openstack_tag_prometheus_io_scrape]
action: keep
regex: 'true'
...# Keep the target if the prometheus_io_address_pool tag is "mgmt" and it matches with the current subnet.
- source_labels:
- __meta_openstack_address_pool
-__meta_openstack_tag_prometheus_io_address_pool
action: replace
regex: "mgmt;mgmt"
target_label: __tmp_keep_target
replacement: "true"
# Keep target if the prometheus_io_address_pool tag is "internal" and it matches with the current subnet.
- source_labels:
- __meta_openstack_address_pool
-__meta_openstack_tag_prometheus_io_address_pool
action: replace
regex: "internal;internal"
target_label: __tmp_keep_target
replacement: "true"
# If you have more subnets, add more relabel configuration stanzas here.# If the prometheus_io_address_pool tag is empty, use "mgmt" by default.
- source_labels:
- __meta_openstack_address_pool
- __meta_openstack_tag_prometheus_io_address_pool
- __tmp_keep_target
action: replace
regex: "mgmt;;"
target_label: __tmp_keep_target
replacement: "true"
# keep only targets that matched any of the above.
- source_labels: [__tmp_keep_target]
regex: true
action: keep

Again this proves how powerful relabeling configuration can be!

Thanks to the great RobustPerception team for all the inspiration on their blog!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store