Collecting system metrics allows system administrators to monitor available resources, detect bottlenecks, and make informed decisions about servers and projects.
At Next Tech, we use system metrics we gather for many things, such as:
- Detecting an issue, like a server that doesn’t have enough resources.
- Conversely, for identifying areas we could be saving money by reducing a server’s resources.
- Ensuring that all services (like a web server or database) are running.
In this tutorial we will go over how to install and configure collectd, which is an open source daemon that collects system performance statistics and provides ways to store and publish them. Then, using the
write_http plugin, we will send our metrics data to a Flask application using HTTP POST requests.
Let’s get started!
Step 1: Load a Python environment
The easiest way to jump into a Python sandbox is using the Next Sandbox, which gives you access to an online computing environment in a couple of seconds. You can click here to launch one, or here to read about how you can install Python on your computer.
Step 2: Installing collectd
The first step is to install
collectd. You can do this by entering the following commands in your terminal:
apt-get install collectd
collectd is installed successfully, move on to the next step — configuring the daemon!
Step 3: Configuring collectd
We need to configure
collectd so that it knows what data to collect and how to send the values collected.
collectd configuration file can be found at
etc/collectd/collectd.conf. If you know Vim, you can modify the configuration file directly by running:
Otherwise, we will link the config file to one that is easily editable in the Sandbox. To do so, run the following three commands in your terminal to link the original configuration file to our new one:
cp /etc/collectd/collectd.conf /root/sandbox/collectd.conf
ln -s /root/sandbox/collectd.conf /etc/collectd/collectd.conf
Now, open the file and take a look at the default
collectd configuration. There are four sections in this file: Global, Logging, LoadPlugin Section, and Plugin Configuration.
The first part of the file displays the Global Settings. The lines beginning with a hash (
#) are commented out — we will remove some of these hashes so that these settings are as follows:
The next section of the configuration file displays plugins used for logging messages generated by the daemon when it is initialized and when loading or configuring other plugins.
For each plugin in this section (and the next), there is a
LoadPlugin line in the configuration, followed by the plugin's options. Almost all of these lines are commented out in order to keep the default configuration lean.
Only one log plugin should be enabled. We will be using using
Remove the hash before the
LoadPlugin logfile line and edit the plugin configuration to write the output to this file by changing the
File parameter. This section should look like this:
Make sure the default logging plugin
syslog is either removed or commented out.
The next section displays a list of features. By default the following plugins are enabled:
collectd daemon will automatically collect data using these plugins. A full list of the available plugins and a short description of each can be found here.
We also want to enable the
write_http plugin so that
collectd will know where to send the data it collects:
Find this plugin in the list and remove the hash to enable it.
The final section shows the configuration for all the listed plugins. You can see all the plugin options in the collectd.conf(5) manual.
Find the plugin configuration for
write_http. We will modify this to look like the following:
Note that we specified the format of our output to be in JSON.
We will not be modifying any of the other plugin configurations but feel free to do so on your own.
Step 4: Verifying configuration
It is important to restart
collectd whenever the configuration file is changed. Run the following command in your terminal to do so:
systemctl restart collectd
You can also run this command to check the status of
systemctl status collectd
If all is working, you should see
Active: active (running) in the output.
You should also be able to see that
collectd was initialized and plugins were loaded in your
Finally, we can verify whether there are issues in the configuration file by running the following:
collectd -t ; echo $?
This command tests the configuration, then exits. It should return the output
Step 5: Creating a Flask app
configd to track our metrics properly...now let's create a web application using Flask so
collectd can send this data via HTTP POST requests!
Flask is a powerful microframework for creating web applications with Python. It comes with an inbuilt development server and is a perfect framework to build RESTful web services. The route decorator which helps to bind a function to a URL can take the HTTP methods as arguments that pave a way to build APIs in an ideal manner.
First, let’s install Flask by running the following:
pip3 install Flask
Now, create a new file called
There are three steps to write this program:
- Create a WSGI application instance, as every application in Flask needs one to handle requests.
- Define a route method which associates a URL and the function which handles it.
- Activate the application’s server.
Copy the following code into the
flask_app.py file in your directory:
This snippet executes the first two steps — we created a WSGI application instance using Flask’s
Flask class, and then we defined a route which maps the path '
/' and the function
get_data to process the request using a Flask's decorator function
Flask.route() decorator, we specified the request methods as
POST. Then, our
get_data method prints the incoming request data using
Continue on to the final step in our lesson to activate the application’s server to see our data!
Step 6: Running a Flask app
To run your Flask application, you need to first tell the terminal what application to work with by exporting the
FLASK_APP environment variable:
Then, execute the following to enable the development environment, including the interactive debugger and reloader:
Finally, we can run the application with the following:
After the app start running, in your terminal you should see your
collectd data coming in as a JSON output!
In this tutorial we covered:
- Configuring multiple
- Building a basic Flask application
- Receiving data from
collectdinside of Flask
We hope you enjoyed this tutorial and learned something new! If you have any comments or questions, don’t hesitate to drop us a note below.
This tutorial was extracted from a lesson on Next Tech. If you’re interested in exploring the other courses we have, come take a look!