From Code to Dashboard: Monitor Your Python Application With Telegraf, Prometheus And Grafana

While the thrill of building and deploying an application is undeniable, it’s the continuous journey of refinement that sets great developers apart from the good. Monitoring is like giving your app a voice. It tells you when it’s feeling overwhelmed, where it’s most engaged, or if it’s encountering any roadblocks. By listening to this voice, you can preempt issues, optimize user experience, and ensure your app stays in its best form. Think of it this way: If development is about creating the heart of your app, monitoring ensures that the heart keeps beating efficiently and effectively.

Photo by Joan Gamell on Unsplash

Why To Monitor Your Application?

When you keep an eye on your app, you catch problems before they get big, learn about your users, and see where things can be better. Monitoring helps make sure your app runs smoothly and stays useful. With tools like StatsD, Telegraf, Prometheus, and Grafana, it’s easier than ever to keep tabs on your Python applications. This is a simple step-by-step guide that will give you all the tools you need to start monitoring your applications!

Kickstarting the Monitoring Journey

When venturing into new technologies or experimenting with fresh ideas, my initial step is always to establish a local setup. This hands-on approach allows me to delve into the intricacies firsthand. To kick things off, you’ll need a couple of essential tools:

• A Python IDE
• Docker
• Docker-Compose

Let’s start our journey of monitoring!

We begin this monitoring journey! We’ll start by constructing a basic FastAPI application that keeps track of how often an endpoint is accessed. To set the stage:

1. Initialize a Python virtual environment (venv).
2. Craft a requirements.txt file with the following entries:

fastapi==0.103.2
uvicorn==0.23.2
statsd-telegraf==3.2.1.post1

1. Execute pip install -r requirements.txt.

This process installs:

• uvicorn, a leading ASGI server for Python.
• FastAPI, a sleek, high-speed web framework tailored for crafting Python APIs.
• StatsD, a proficient statistics aggregator.

Building the FastAPI Application

Now let’s create a main.py file and build our API app:

from fastapi import FastAPI
import uvicorn

app = FastAPI()


@app.get("/")
def read_root():
    return {"Hello": "World"}


if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

Navigate to the project’s root directory in your terminal. Execute the command python main.py. Voilà! Your API should spring to life, and you'll be greeted with the following:

API Is Up And Running

To confirm our API is live and responsive, execute the following command in your terminal:

curl localhost:8000

You should receive a response that reads:

{"Hello": "World"}

With our API operational, it’s time to enhance it with custom metrics. Begin by crafting a new file named metrics.py

We’ll begin by establishing a foundational metrics structure. Create a simple Python class named APIMetrics with the following content:

class APIMetrics:
    REQUEST_COUNT = "api.requests.count"

This sets the stage for our REQUEST_COUNT metric.

Now, let’s refine our main.py to incorporate these metrics:

from fastapi import FastAPI
import uvicorn
import statsd
from metrics import APIMetrics

app = FastAPI()
metrics_client = statsd.StatsClient(host="telegraf", port=8125)

@app.get("/")
def read_root():
    metrics_client.incr(APIMetrics.REQUEST_COUNT)
    return {"Hello": "World"}


if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

In the provided code snippet, we’ve integrated a custom metric into our FastAPI application. Using the statsd.StatsClient, we're set to log metrics to a Telegraf instance listening on port 8125. Specifically, with the line metrics_client.incr(APIMetrics.REQUEST_COUNT), we're incrementing a metric named REQUEST_COUNT from our APIMetrics class every time the root endpoint ("/") of our API is accessed or pinged. In simple terms, each time a user or service pings our API, this metric gets logged, allowing us to monitor and gauge the frequency of access to our application's primary endpoint.

Seamless Integration with Docker, Telegraf, and Prometheus

The magic unfolds as you containerize your application and set up Telegraf and Prometheus for precise monitoring. This involves crafting configuration files, defining ports, and ensuring smooth data transition between the tools.

To containerize our Python application, let’s craft a Dockerfile:

FROM python:3.11.6-slim
WORKDIR /code

COPY ./requirements.txt /code/requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

COPY ./ /code/

Lastly, for seamless integration, we’ll need configuration files. These guide Telegraf and Prometheus on the specific ports to monitor and designate data output paths:

telegraf.conf :

[[inputs.statsd]]
  protocol = "udp"
  max_tcp_connections = 250
  tcp_keep_alive = false
  service_address = ":8125"
  delete_gauges = true
  delete_counters = true
  delete_sets = true
  delete_timings = true
  percentiles = [50.0, 90.0, 99.0, 99.9, 99.95, 100.0]
  metric_separator = "_"
  parse_data_dog_tags = false
  datadog_extensions = false
  datadog_distributions = false
  allowed_pending_messages = 10000
  percentile_limit = 1000
  
[[outputs.prometheus_client]]
  ## Address to listen on.
  listen = ":9273"

prometheus.yml :

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
scrape_configs:
- job_name: scrape_telegraf
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - telegraf:9273

It’s now time to launch our local ensemble: the API, Telegraf, Prometheus, and Grafana.

Begin by crafting a docker-compose.yml file with the content below:

services:

  api:
    build: .
    command:
      - python
      - main.py
    ports:
      - 8000:8000

  telegraf:
      image: telegraf
      volumes:
        - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
      ports:
        - 9273:9273

  grafana:
    image: grafana/grafana
    ports:
      - 3000:3000

  prometheus:
      image: prom/prometheus
      container_name: prometheus
      command:
        - '--config.file=/etc/prometheus/prometheus.yml'
      ports:
        - 9090:9090
      restart: unless-stopped
      volumes:
        - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

Great progress! Execute the command docker-compose up -d --build. Once everything is initialized, navigate to localhost:3000 in your web browser. This will lead you to Grafana's login page. Simply enter adminfor both the username and password fields, and you'll be greeted with the main dashboard.

To link Prometheus to Grafana, follow these steps:

1. Navigate to Menu -> Connectors -> Data Sources.
2. Click Add Data Source and select Prometheus.
3. In the ‘Prometheus Server URL’ field, input http://prometheus:9090.
4. Finally, scroll down and select ‘Save & Test’.

Upon successful connection, a green confirmation message will appear, signifying Grafana’s successful linkage to Prometheus.

Simulating Real-World Traffic

Experience the system in action! Use commands to mimic real-world traffic to your application, witnessing in real-time how your application interacts, performs, and logs metrics.

Head back to your terminal and execute the command:

while true; do curl localhost:8000; sleep $((RANDOM % 6)); done

This continually pings our API at random intervals, mimicking real-world traffic to our application.

After waiting a brief moment, navigate to Grafana’s ‘Explore’ view. Enter api_requests_count into the Metrics Browser. Voilà! You should now witness the traffic statistics for our API:

You’ve successfully set up a comprehensive monitoring system with StatsD, Telegraf, Grafana, and Prometheus. While this is just the beginning, it offers a glimpse into the workings of these technologies and provides a solid foundation from which to delve deeper.

Let’s Summarize

Building and deploying an application is only part of the developer’s journey. Monitoring its performance and understanding its real-time behavior is pivotal to its success. In this article, we’ve walked through the steps to set up a local FastAPI application, equipped it with custom metrics, and then leveraged the power of Docker, Telegraf, Prometheus, and Grafana for comprehensive monitoring. We’ve touched upon containerizing the Python app, integrating Prometheus with Grafana, and simulating real-world traffic to witness these tools in action. Dive in to discover the importance of monitoring and the seamless integration of these tools for an efficient, informed, and proactive application development approach.

Co-authored by Ofek Katriel, this article dives deep into the intricacies of monitoring Python applications for optimal performance.