Securing Prometheus between your Prometheus Instance and your Polkadot / Kusama Validators

Maintaining a secure and robust monitoring infrastructure for your validator nodes is a critical component of providing validator services to the Polkadot community. If you are using Prometheus for monitoring, your deployment should ensure that the connectivity between your Prometheus instance and your scraping targets (validator nodes) is secure. What follows is detail on exactly how to do that.

If you run the W3F Secure Validator (SV) setup, you may have noticed the following two parameters in the Ansible inventory setup files:

# Nginx authentication settings.
nginx_user='prometheus'
nginx_password='nginx_password'

Let’s dive deep into this and see exactly what’s going on so that this can be deployed even if you are not using the W3F setup.

NGINX, the reverse proxy used in the SV setup, allows for the configuration of basic HTTP authentication. You can see the steps for configuring this in the NGINX documentation, here. Fortunately for us, W3F has built that all into an Ansible playbook. The sample inventory has a configuration for this with ‘prometheus’ as the username, and ‘nginx_password’ as the password, shown above.

The actual setup of this involves encrypting the NGINX username and password, and then adding it to the NGINX configuration on your validator node. You can also take this a step further if you’ve forked the W3F SV repo by configuring NGINX to filter based on the source-IP of your Prometheus instance IP, further restricting access to the service.

An example of the source-IP filtering configuration is below, copied from the NGINX docs site:

location /api {
    #...
    satisfy all;    

    deny  192.168.1.2;
    allow 192.168.1.1/24;
    allow 127.0.0.1;
    deny  all;

    auth_basic           "Administrator’s Area";
    auth_basic_user_file conf/htpasswd;
}

Whether you are using the SV playbook, or have deployed this separately, that’s only one side of the configuration. Next, you will need to tell your Prometheus instance that it will use a username and password when it goes to scrape data from you validator. This requires a simple addition to each job you have in your prometheus.yml. An example of this is below:

- job_name: 'substrate'
    scrape_interval: 5s
    static_configs:
      - targets: ['8.8.8.8:9616']
        labels:
          instance: 'Google-DNS-On-Substrate'
    basic_auth:
      username: NGINX-Prometheus
      password: MyComplexPassword

And that’s it, once you restart your services, your Prometheus instance should now be authenticating against your validators to gather data. If we want to take this another step further, we can implement TLS encryption between the Prometheus instance and validator node using self-generated certificates, and that will be the topic of a future article.