Leveraging Promtail and Loki to get more insights of NGINX
When it comes to monitoring, you want to get every piece of important information that is available and store it somewhere so we can query and visualize it later on a dashboard.
There are a lot of ways to extract data from a server/node/instance. One way is through a Prometheus exporter. There are a lot of Prometheus exporters which are open source and available on Github and other platforms. Another approach is collecting, parsing and analyzing logs.
In this blog post I will focus on the second approach: extracting data from an instance using Promtail and Loki.
The problem with exporters (sometimes)
Most of the Prometheus exporters are good and provide you with all the metrics you need to build a good monitoring/alerting system.
But sometimes this is not the case, in this blog post I am going to use this exporter as an example: https://github.com/nginxinc/nginx-prometheus-exporter
As you maybe can see in the README of the project, there are not a lot of metrics available in the NGINX OSS version of the exporter. Only a handful of metrics are exposed and most of these metrics do not provide useful information for our purposes.
Solving this problem
In this example, the NGINX access and error logs contain a gold mine of information which we can translate into more useful metrics.
That is where Promtail and Loki comes into play. Promtail can ship the contents of local logs to a private Loki instance. In this example I am using this dashboard setup: https://grafana.com/grafana/dashboards/13865. Here you can follow along on how to setup the required NGINX json log format configuration and the Promtail config.
This setup gives us more flexibility on:
- which information we want to select
- how to parse and label this information
This gives us a better understanding on how NGINX is performing and what is being served.
Conclusion
Let me be clear, I am a big fan of Prometheus exporters, but sometimes they just don’t provide enough information to make an insightful dashboard.
This is where Promtail and Loki can be your extending tool to achieve this. The possibilities in parsing logs are endless and can be extremely powerful in achieving more intel of how well your component is performing.
