Grafana & Prometheus Fundamentals

What will we learn, and what will we try?

1 — What is the Grafana?
2 — What is the Prometheus?
3 — How to use Grafana and Prometheus together.
4 — Fundamental Grafana and Prometheus Example

1 — What is the Grafana?

Grafana is an open-source analytics and visualization platform that enables users to collect, analyze, and understand data from various sources by creating real-time and interactive visualizations. It supports integration with different data sources (e.g., Prometheus, Elasticsearch, InfluxDB, MySQL, Azure Monitor, etc.) and allows users to gather, analyze, and comprehend data through charts, dashboards, and reports.

2 — What is the Prometheus?

Prometheus is an open-source system and application monitoring tool designed for modern application development and deployment environments. Developed by the Cloud Native Computing Foundation (CNCF), Prometheus focuses on the complexity of applications running in dynamic environments, such as microservices architectures and container technologies.

Graphing and Visualization: While Prometheus offers basic graphing and visualization tools, more comprehensive visualizations are often achieved by integrating with external tools. Tools like Grafana are commonly used to create impressive and meaningful graphs using data retrieved from Prometheus. Prometheus can alert users when specific conditions are met. Predefined alerting rules can be created, and these alerts can be directed to various communication channels (email, Slack, Webhook, etc.).

With these features, Prometheus provides a scalable and flexible monitoring solution designed to adapt to the complexity of modern application infrastructures.

3 - How to use Grafana and Prometheus together.

Prometheus collects and stores metrics. Grafana, in addition to Prometheus, can also receive data from other sources. Grafana can visually present the received data in interactive dashboards and graphs. Prometheus monitors specific conditions and can generate alerts. Grafana can receive these alerts and convey them to users through visual notifications or other channels.

When used together, Prometheus and Grafana provide a comprehensive solution for system monitoring and performance analysis. Prometheus collects and stores metrics, while Grafana visually presents these metrics in an intuitive manner. This allows system administrators and developers to monitor the status of applications and infrastructures more effectively.

4 — Fundamental Grafana and Prometheus Example

Grafana installation:

brew update
brew install grafana

To start Grafana, run the following command:

brew services start grafana

Go to the Grafana panel: http://localhost:3000/

Prometheus installation:

brew install prometheus

To start Prometheus, run the following command:

brew prometheus start

Go to the Grafana panel: http://localhost:9090/

Prometheus Configuration: Edit the prometheus.yml configuration file to scrape metrics from your web application. Here’s a basic example:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'app'
    static_configs:
      - targets: ['localhost:8092']

Ensure your web application exposes metrics in Prometheus format. You can use a library like Prometheus client libraries for different programming languages. Below is an example in Python using the prometheus_flask_exporter library.

from flask import Flask
import random
import time
from prometheus_flask_exporter import PrometheusMetrics

app = Flask(__name__)
metrics = PrometheusMetrics(app)


@app.route('/')
def hello_word_page():
    time.sleep(random.uniform(0, 1))
    return "Hello, World!"


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8092)

Create and run this file.

Grafana Configuration:

1. Start Grafana and access it through your web browser (http://localhost:3000 by default).
2. Log in with the default credentials (admin/admin).
3. Add Prometheus as a data source:
   Go to “Settings” > “Data Sources” > “Add your first data source.”
   Choose Prometheus and enter the URL (http://localhost:9090 by default).

4. If you need to update the URL, You can change it.

5. Add Grafana dashboard

6. Select configuration.

7. Select Metric. For example, flask_http_request_duration_seconds_count and use query.

Final Steps! Show Your request Count in the Grafana dashboard.

Visit the Hello Word URL and check the Grafana dashboard. If you correct anything, you can apply and save.

Where do we use Grafana and Prometheus?

In our live automation, we utilize Grafana and Prometheus to log our failed test cases and visualize the rates of flaky, skipped, and passed cases. We also use them to identify how many of these cases failed on the created machines, view failed automation cases for teams within Insider, and log failures in automation tasks.

In Jira, we use Grafana and Prometheus to see the count of bugs found by QA in opened bug tasks and to determine the average bug count for developers. Additionally, we use them to find the average number of bugs discovered by QA.
We log details such as the average runtime, memory usage, and CPU usage of our automations, as well as the types of machines on which automations run.

Bug task metrics such as SLAs, statuses, urgencies, and many others are logged and monitored.
In our Kubernetes environment, we log and monitor pod counts, node storage, memory, and CPU usage.
Daily or hourly job data, running clusters, and the duration of active pods are logged and monitored.

We monitor the system’s hourly, daily, weekly, and monthly costs and efficiency.
Metrics such as the types of incoming requests (Get, Post, Update, Delete, etc.), input and output sizes, latency, and similar data are monitored.
For APIs, we monitor total requests, responses, statuses, sizes, and latencies.

Conclusion

In conclusion, Prometheus and Grafana form a powerful duo in the realm of system monitoring and observability. Prometheus, with its robust metric collection and storage capabilities, serves as the backbone for capturing crucial performance data from various sources. Meanwhile, Grafana excels in transforming this data into visually compelling dashboards and graphs, providing a user-friendly interface for analyzing and understanding system behavior. The seamless integration between Prometheus and Grafana offers a comprehensive solution for real-time monitoring, alerting, and performance analysis, making them essential tools for system administrators and developers alike.

Finally, I’m pleased to be a part of the Insider and very excited about the work we will do in the future. If you would like to learn SOLID Principles, you can visit https://medium.com/insiderengineering/solid-principles-15d335a1f8a6.

For any questions, you can reach me at the email address provided below: https://www.linkedin.com/in/umut-yasin-çolak/