How To Set Up Prometheus To Monitor Your Ubuntu Server
It was designed by SoundCloud as a tool to monitor their systems, and features a time series database with multi-dimensional data storage coupled to a powerful query language to enable system administrators to define their metrics and generate reports.
Prometheus consists of a couple of different elements, the Prometheus server itself and the Node Exporter.
Getting Started With Prometheus
Unfortunately whilst Prometheus is included in the repositories for Ubuntu 16.04, as that isn’t released yet we’ll need to be installing it from the binary releases on GitHub. So first we’ll need to get the release from the Prometheus GitHub repository and extract it.
tar -xvzf ../prometheus-0.17.0.linux-amd64.tar.gz
This then creates a directory named after the archive with the Prometheus server files inside. So we’ll need to change to this directory to test it.
Given that everything went fine so far you should have an output from Prometheus with it’s version and the version of Go it is using. As Prometheus was designed with the primary purpose of monitoring web services it doesn’t natively monitor your local systems. This is where the Node Exporter part comes in, this takes the information from your system and passes it into Prometheus. So the next step is to download and install the Node Exporter:
tar -xvzf ../https://github.com/prometheus/node_exporter/releases/download/0.11.0/node_exporter-0.11.0.linux-amd64.tar.gz
Now once started the node exporter will run in the background and export the metrics to a web interface that can be used to capture them. To make it tie in with how services are normally handled on Ubuntu we can create an Upstart script to start the service at bootup, so first create a link for the node exporter in /usr/bin
sudo ln -s ~/Prometheus/node_exporter/node_exporter /usr/bin
Next we need to create the upstart script, so open it as below, note that I’m using nano to edit the file here, other text editors are available:
sudo nano /etc/init/node_exporter.conf
Then paste the following lines into that file:
#Prometheus Node Exporter Upstart script
start on startup
That was a nice simple script so save and exit and then the service command can now be used to start and stop the node exporter as required.
service node_exporter start
You can check that it works by connecting to your server on port 9100 with a web browser and looking at the metrics file, this will display loads of metrics about your server that Prometheus will log.
Now the next thing to do is get Prometheus listening to the metrics from the node exporter, for this the first thing that is needed is to create a configuration file for Prometheus.
sudo nano /etc/prometheus.yml
Now add the following config:
- job_name: “node”
- targets: [’localhost:9100’]
Save and exit the file and all should be ready to go. We’ve called the job name here “node” as that picks up some default configuration for the node exporter and saves a lot of time filling in the configuration file.
Testing If Monitoring Is Working
Now the moment of truth, if you start Prometheus it should pick up the config file and start up without any errors. So that’s easily done by running the main Prometheus executable:
prometheus -config.file /etc/prometheus.yml
You should see some status messages scroll up the screen, the important ones to note are that there’s one that says “Loading configuration file prometheus.yml”, “Target manager started.” and “Listening on :9090”. If you can see all of those and there are no errors then all should be working well. You should now be able to connect to port 9090 on your server to see your metrics.
With all this working you can then use ctrl+c to stop Prometheus from running as logging in to your server and manually running it every time the server reboots isn’t very practical. This means the next step is to create an upstart file for Prometheus itself. First we’ll create a link for Prometheus in /usr/bin as before:
ln -s ~/Prometheus/server/prometheus-0.17.0.linux-amd64/prometheus /usr/bin
Then open the upstart configuration file for Prometheus:
With the Upstart configuration file open, paste in the following text.
#Prometheus Upstart Script
start on startup
/usr/bin/prometheus -config.file /etc/prometheus.yml > /var/log/prometheus.log 2>&1 &
Now you can start Prometheus as a service:
service prometheus start
Output from Prometheus is now logged to the /var/log/prometheus file, and Prometheus is now running in the background logging your server’s metrics, it will also start up when your server reboots. If you browser around the Prometheus web interface you should see it monitoring your server’s status and also a selection of graphs detailing showing that data.
Never miss another post. Sign up for the weekly 100TB newsletter or follow us on Facebook and Twitter.
Originally published at blog.100tb.com.