Monitoring you Golang server with Prometheus and Grafana
I recently became very interested on how you can learn insights from your application by monitoring it. Once I started searching about monitoring tools, it didn’t take too long to find out about Prometheus, which will collect metrics from your application, and Grafana, which will display the collected metrics.
How to reproduce this tutorial and run the code
To make it easy to reproduce this tutorial, we will use Docker, Docker-Compose and VS Code Remote Container. I will not go into details on how to install those tools, but by simply following the documentation you should be fine.
You can find the full source code here.
Open the source code on VSCode and then click on Reopen in Container. Once the initialisation is finished, you should have three containers running under the service name of go-prometheus-grafana. You can now start the application by the GUI of VSCode or running the following command on the terminal: go run main.go
The user_status custom metric and the data being generated
Once you start the application (specifically, the main.go file), it will create a custom counter metric called http_request_get_user_status_count and register it on the prometheus client. Both server and producer will be initialised. The producer will send messages to the server containing a random user name. The server then generates a random status for that request and then adds the response to the prometheus counter. We register an additional endpoint for Prometheus under the path “/metrics”. The code can be seen below:
Prometheus
During initialisation, the prometheus service was configured to scrape for data every 5s on the “/metrics” path having the server’s name as the target (since all services are running under the same nertwork, each service can discover the other by using the other service’s name):
Prometheus should be running on http://localhost:9090. On the search expression, if you type http_request_get_user_status_count, you should see the data generated so far. Something like:
Grafana
Now, to visualise the data in Prometheus, we will use Grafana. Grafana should be running on http://localhost:3000. The default credentials are: admin/admin.
The first thing we need to do is to add prometheus as a data source. To do so, on the left side panel, click on the gear icon and then on Data Sources and finally Add Data Source. Select Prometheus and on the URL, enter: prometheus:9090, like the following image:
Scroll down, and click on Save and Test, you should see a message saying: “Data source is working”.
Now, on the left side panel, click on the “+” icon and then Import. On the new page, copy and paste the below json (or upload the file grafana/panel.json from the source code) and click Load.
You should see something similar to the below on the screen:
It should be all. Of course, this is a extremely over simplified example that I used to understand how all parts connect to each other. Again, you can find the full code here:
