Instrumenting phoenix with Prometheus
Why?
- Instrumentation is about collecting metrics for your application. You can collect database query time, response time, basically anything you want. If you have a particular thing to measure, prometheus elixir package makes it really easy to collect metrics for your phoenix app.
- Even if there isn’t anything in particular you want to measure, some prometheus adapters provide you with more granular metrics out of the box (Ecto’s query’s average time, controller average response time, view average render time…)
In this small article We will cover the installation of the prometheus adapters on your phoenix app and the deployment with docker.
We will be
How
add the following dependencies to your mix.exs
{:prometheus, "~> 4.0", override: true},{:prometheus_ex, "~> 3.0"},{:prometheus_ecto, "~> 1.0"},{:prometheus_phoenix, "~> 1.2"},{:prometheus_plugs, "~> 1.0"},{:prometheus_process_collector, "~> 1.3"},
(here we are using override just to use the latest version of prometheus itself)
in your config/config.exs
add
# under your endpoint configconfig :my_app, MyAppWeb.Endpoint,
# leave your current settings here, just add the following under
instrumenters: [MyApp.PhoenixInstrumenter]
config :prometheus, MyApp.PhoenixInstrumenter,controller_call_labels: [:controller, :action],duration_buckets: [10,25,50,100,250,500,1000,2500,5000,10_000,25_000,50_000,100_000,250_000,500_000,1_000_000,2_500_000,5_000_000,10_000_000],registry: :default,duration_unit: :microsecondsconfig :prometheus, MyApp.PipelineInstrumenter,labels: [:status_class, :method, :host, :scheme, :request_path],duration_buckets: [10,100,1_000,10_000,100_000,300_000,500_000,750_000,1_000_000,1_500_000,2_000_000,3_000_000],registry: :default,duration_unit: :microsecondsconfig :my_app, MyApp.Repo, loggers: [MyApp.RepoInstrumenter, Ecto.LogEntry]
MyApp.RepoInstrumenter
, MyApp.PhoenixInstrumenter
and MyApp.PipelineInstrumenter
correspond to three modules that we will create next (of course you can name the modules whatever you want). I put these modules under myapp/instrumenters/repo_instrumenter
for example, but that is also a matter of taste.
add a myapp/instrumenters/repo_instrumenter.ex
with the following
defmodule MyApp.RepoInstrumenter do@moduledoc """Instrumentation of Ecto activity."""use Prometheus.EctoInstrumenterend
add a myapp/instrumenters/phoenix_instrumenter.ex
with the following
defmodule MyApp.PhoenixInstrumenter do@moduledoc """Provides instrumentation for Phoenix specific metrics."""use Prometheus.PhoenixInstrumenterend
add a myapp/instrumenters/pipeline_instrumenter.ex
with the following
defmodule MyApp.PipelineInstrumenter do@moduledoc """Instrumentation for the entire plug pipeline."""use Prometheus.PlugPipelineInstrumenter@spec label_value(:request_path, Plug.Conn.t()) :: binarydef label_value(:request_path, conn) doconn.request_pathendend
now in myapp/application.ex
add
alias MyApp{
PhoenixInstrumenter, PipelineInstrumenter, PrometheusExporter, RepoInstrumenter, VersionInstrumenter}#! add these inside the start function (anywhere is fine)require Prometheus.RegistryPhoenixInstrumenter.setup()PipelineInstrumenter.setup()RepoInstrumenter.setup()if :os.type() == {:unix, :linux} doPrometheus.Registry.register_collector(:prometheus_process_collector)endPrometheusExporter.setup()
Setting up prometheus and grafana
prometheus will collect the metrics, but you need something to visualize them. Let’s use grafana and let’s set everything up with docker.
if you haven’t set up your deployment with docker, check this for help.
once you have a docker-compose.yml
file, add the following under the services list
prometheus: image: prom/prometheus ports: - "9090:9090" volumes: - ./prometheus/:/etc/prometheus/ - prometheus_data:/prometheus command: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.path=/prometheus' - '-- web.console.libraries=/usr/share/prometheus/console_libraries' - '--web.console.templates=/usr/share/prometheus/consoles' restart: alwaysgrafana: image: grafana/grafana:5.1.0 user: "104" volumes: - grafana_data:/var/lib/grafana - ./grafana/provisioning/:/etc/grafana/provisioning/ links: - prometheus ports: - "3000:3000" restart: always
add a prometheus/prometheus.yml
file on your local with (this is just the default prometheus config file, feel free to customize)
global:scrape_interval: 15s # By default, scrape targets every 15 seconds.# Attach these labels to any time series or alerts when communicating with# external systems (federation, remote storage, Alertmanager).external_labels:monitor: 'codelab-monitor'# A scrape configuration containing exactly one endpoint to scrape:# Here it's Prometheus itself.scrape_configs:# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.- job_name: 'prometheus'# Override the global default and scrape targets from this job every 5 seconds.scrape_interval: 5sstatic_configs:- targets: ['web:4000']
so basically we are creating a volume to store the prometheus config and we are linking prometheus and grafana services together. Here is how your docker-compose file should look like.
verify that your docker setup is working with docker-compose up
you can go to localhost:3000
in your browser and should see the grafana interface. Default user and password is admin (for both).
Next you need to add a datasource to Grafana. On the left, on the settings icon, click on datasource to add a new datasource. Name it whatever you want (ex MyApp Instrumentation). choose prometheus as the source type. Use http://prometheus:9090 as the url. And click on save & test. It should confirm that the setup is correct.
Next you need to add a dashboard. You can create your own dashboard, but let’s use some open source one at first. On Grafana’s interface on the left, click on the + and import. You can directly paste some JSON in there. There is a repo with lots of wonderful dashboards. Three are particularly useful
the BEAM dashboard (you can directly copy and paste the json from the repo into the import part of grafana)
and the elixir dashboard (which is the one we want for phoenix)
These three dashboard will work out of the box with our setup.
The packages we installed, give you a lot more metrics, so you can make different dashboard. Dashboard customization will be for another article :-)
This article is part of the phoenix unofficial guide ( a guide that relates to everything requiring third parties and phoenix (deployement, continuous integration, instrumentation…)). This article will be updated on the github repo here.
based on the really good article written in 2016 (I wrote an article, just because I got a stuck when I tried to apply that article). By the way, if you ever get stuck, the #prometheus channel on elixir slack, is just awesome.