Understanding your Symfony app with Prometheus

You can try all of the examples by downloading this repository https://github.com/rk4xxicom/symfony-prometheus-example (you will need docker-compose to run it though).

If you are already familiar with Prometheus / OpenMetrics format, just skip the introduction and move on to the Symfony part.

Why Prometheus?

In short, almost any web application needs some sort of observability in production.

As developers we need to understand how many resources it actually consumes relative to the traffic it serves. But not just any kind of traffic, we (ideally!) would want to distinguish between different kinds of traffic. For example, buying something on an e-commerce website often requires more resources than just displaying a product page.

Of course, as project stakeholders we are also interested in how many items have been sold this month and not just in the total number of hits.

There are many tools that can solve this sort of problem in your project, but at 4xxi we have found that Prometheus suits our needs — at least for now. It can be easily integrated with both legacy and modern web apps in PHP and Go, has plenty of different integrations for web servers, databases, Linux, message queues and it fits well in a container-based environment.

Eventually, the Prometheus metrics format will become an open standard for metrics collection — OpenMetrics. So even if you find that Prometheus does not work for you, keep in mind that major cloud providers support Prometheus format (DataDog, NewRelic).

A short introduction to OpenMetrics and Prometheus

Prometheus works using the “pull” model over HTTP protocol — give it a URL and it will start scraping it, parsing the metrics, and saving them into internal time-series database. Normally, if you want to start monitoring a system or a product, you run a small utility or an extension that would start an HTTP server and translate the internal state of the product into OpenMetrics format.

Metrics consist of types, labels, and values (either integer or float). Here is a simple metric that was taken from a Linux-level exporter:

# HELP node_filesystem_avail_bytes Filesystem space available to non-root users in bytes.
# TYPE node_filesystem_avail_bytes gauge
node_filesystem_avail_bytes{device="/dev/nvme2n1p1",fstype="ext4",mountpoint="/"} 4.617367552e+09
node_filesystem_avail_bytes{device="/dev/nvme1n1",fstype="ext4",mountpoint="/sftp"} 6.922203136e+09

The important parts here are:

1. TYPE — it’s a “gauge”, basically, a value that can go up and down (which is true for the number of available bytes). If we were to measure the total number of requests since the start of the project, that would be a “counter”, which would allow us to do some neat calculations over it.
2. Labels: devices, fstype, mountpoint — should be self-explanatory. It should be noted that labels can be attached later to simplify charts or distinguish between different instances.
3. The value itself — the exporter reports the system state at the time of scraping. It is however unreadable for a human eye, and you might wonder if you should just install NewRelic agent instead.

However, for the system monitoring tools there are generally some charts that you can install to you Grafana or another dashboard. Here is the one that we used https://grafana.com/grafana/dashboards/1860, and here is how it interprets the values above (in conjunction with another metric, total_bytes):

I did not have to write code for this one

Now, let us move onto Symfony integration and application metrics.

The easy way: just use SQL

If you’re strapped for time or don’t really want to dive into the old code, here is a quick solution that you could try

Suppose you have a number of accounts in different stages and you want to see how that number changes over time — when do you get an influx of new accounts, how it is correlated with CPU load etc. (this example is loosely based on a real project).

Let’s create the Entity class describing our account:

Now we need to get the data from the database:

And finally, we need to translate the data into OpenMetrics format for Prometheus to consume:

Now, if you launch your application, you’ll see something like this

Now we need to tell Prometheus about this.

I won't talk about installing Prometheus here, if you do not have it installed, check out this documentation page or again, try all of the examples for the article in the repository.

What I will talk about, however, is how to add it to a Prometheus configuration file:

global:
    scrape_interval:     15s 
    evaluation_interval: 15s
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
    - job_name: 'legacy_metrics'
      metrics_path: /legacy/metrics
      static_configs:
          - targets: ['yourserverhost:80']

Here we specify the host, the port, and the path from which metrics can be scraped by Prometheus. If you use k8s, there is a better way to tell Prometheus about your services and pods, but we are trying to keep it simple here.

After restarting the Prometheus instance, you should be able to access the web interface by going to http://localhost:9090 and you can check if your changes were applied properly:

Caveats and downsides

• this method could run into performance issues depending on the SQL queries (I did not run into such issues, but it is possible)
• there is no easy way to keep the state between requests (e. g. to show the total number of requests or logins over time)

The downside of this method is that you cannot easily keep the state between different requests, for example, to store the total number of requests or logins to the application.

Slightly harder way: extracting data from logs

If you need to monitor such metrics, but do not want to rewrite the application code because of it, you can try to extract the data from the logs.

Google created a tool called mtail that parses log files and exposes extracted metrics via HTTP endpoint. And yes, it handles log rotation.

Let us try it on a Symfony app:

Here each time a user is created the app puts its id in the logs.

mtail has a simple DSL to translate log lines into metrics:

Put the program in /etc/mtail .

Download a “mtail” binary from Github or build a docker image, and run it specifying logs folder and “progs” folder.

./mtail — logs '/mnt/log/*.log — progs /etc/mtail

This should launch a web server on localhost:3903

Triggering a new log entry

curl -X POST http://localhost/user -d name=testusername

The user total has increased!

Caveats and downsides

• Symfony uses fingers_crossed handler for monolog in production and the info level logs will not actually be written to files most of the time; to extract metrics from logs you will need to write all the relevant logs to the file
• in some systems where log files are replaced by an external tool, mtail becomes confused and re-reads logs multiplying metrics. This should not be an issue in production, but it could be a problem when using docker-sync on MacOS
• mtail depends on log files being actually available and readable as files (i. e. if you send your logs to Papertrail, you’ll need to keep a file for mtail)
• mtail has no in-built authorization mechanism, so it either should be run in a private network or be proxied via Nginx or another web server with SSL and something like basic auth

Despite the issues described above, it’s a useful tool in many cases, when working with legacy apps or apps that are not written by you (e.g. web server — there are some readymade dashboards that display average response time etc. based on logs https://grafana.com/grafana/dashboards/9516)

The right way

If you use Symfony 4/5, there is a great Symfony bundle for Prometheus.

Metrics are stored either in apcu or in Redis.

The README has everything you need to set it up, I’ll just provide a simple example.

After enabling the bundle via flex or using README, add collector registry as a dependency.

In the same place where we added logging before, increment a counter manually.

“getOrRegisterCounter” adds a new counter, with relevant parameters being the “name” of the counter, “help”, and the last one — the list of labels.

After that, we increase the counter twice — first with type “all” and then with type “enabled” (because users are enabled by default).

If we were to disable a user, we would increment the same counter with type “disabled”.

Let us see how it works! Create a user one more time:

curl -X POST http://localhost/user -d name=testusername

and go to http://localhost/metrics/prometheus

Metrics are stored in Redis as hashes. You can see the exact data that is being stored by connecting to Redis:

Since it is working now, we once again need to tell Prometheus what to scrape (the full config can be found here).

I created 26 new users by calling the endpoint at random periods of time and here is the result

Same data represented as “new users per minute”:

You can find more info on how to work with counters and rates here and here.

The bundle also gives you some application metrics, such as the number 500x and 200x responses, percentiles of response times for different routes, etc.

Caveats and downsides

• due to the nature of most PHP web apps, you cannot just keep the metrics in memory of the application, and you will need to install external storage such as Redis or APC
• response times presented by the bundle are collected from inside the application, so if there is an issue between Nginx and php-fpm for example, it will not be reflected in the metrics — but it is a good start if you do not have any other sources of monitoring data yet.

Conclusion

Observability of application and business metrics is often underestimated compared to technical metrics such as “number of hits“, “cpu load” and others.

To better understand your application and how it can grow, you need a way to track some higher-level metrics.

One such solution is Prometheus which is quickly becoming a standard in the industry (although there are many other great tools).

If you have a Symfony application, either a legacy or an actively developed one, there is an easy way to start monitoring and learning.

Here are some useful links if you’re interested in monitoring:

• https://www.prometheusbook.com (a concise and practical book about monitoring your whole stack)
• http://www.brendangregg.com/usemethod.html (and the blog is general is very insightful, although directed more towards DevOps side)
• https://landing.google.com/sre/sre-book/chapters/practical-alerting/ (Google’s internal system Borgmon served as an inspiration for Prometheus)