Application Monitoring with Prometheus and Grafana (Part 3)

Patel Romil
Jun 27 · 4 min read

In this series, we have created the client applications with REST APIs and used Spring Boot Actuators to monitor the health of the applications, metrics for the REST APIs, Database Connections & Status, Tomcat and Hikari Connections, Cache Management, Scheduled Tasks, Environment of an application and much more. To visualize and monitor the applications we need UI, to achieve this we have created Spring Boot Admin which acts as a server application and uses JSON information from Actuators and provides visualization in form of graphs and tables.

Prometheus

Prometheus is an open-source system that provides a more flexible and great visualized information, alert mechanism, rules, powerful queries, storage, etc. Most Prometheus components are written in Go, making them easy to build and deploy as static binaries.

Image for post
Image for post
Architecture by Prometheus

Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be used to visualize the collected data.

Let’s download the Prometheus with OS like windows and architecture as amd64. Here I have used the prometheus-2.16.0.windows-amd64. Extract it at an appropriate place and you can see prometheus.yml where we can customize the Prometheus with scarp intervals, alert managers and rules, etc. We can also specify which configuration file to load, using --config.file flag.

Before going ahead, make sure we have done the required configurations in our application.

  • Include spring-boot-starter-actuator, micrometer-registry-Prometheus in our pom.xml
  • /prometheus endpoint is exposed in application-properties
  • We can use enable and expose configurations with Spring Security to prevent unauthorized access to sensitive information.
  • metrics_path and scrape_interval is configured correctly in prometheus.yml
pom.xml
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
application-properties
#Include end-points
management.endpoint.metrics.enabled = true
management.endpoints.web.exposure.include = *
management.endpoint.metrics.show-details = always
management.endpoint.health.show-details=always

Grafana

Grafana is the most popular open-source project that allows you to query, visualize and alert on metrics and logs no matter where they are stored. It uses time-series databases like Graphite, Prometheus, Elasticsearch, OpenTSDB and InfluxDB to query. We can also integrate Grafana with Splunk, AppDynamics, DataDog, and support Custom Auth (LDAP, OAuth, and more), SSL and Custom Domains (enterprise version). For visualization, we can use ready to use dashboards provided by it. You can download the latest version from here.

Before going ahead, we have to add a data source in Grafana and can create the dashboard for application monitoring and visualization.

Image for post
Image for post
Add DataSource

Let’s Run & Monitor

  • Prometheus can be accessed on localhost:9090
  • Grafana can be accessed on localhost:3000
Image for post
Image for post
Prometheus: localhost:9090/graph
Image for post
Image for post
Grafan: localhost:3000
Image for post
Image for post
Grafan: localhost:3000
Image for post
Image for post
Grafan: localhost:3000
Image for post
Image for post
Grafan: localhost:3000
Image for post
Image for post
Grafan: localhost:3000

Rules & Alert Mechanism

In Prometheus and Grafana, we can configure the alert mechanism and rules according to the requirements and take immediate actions based on health, load, and connections of the instances.

# Rule files specifies a list of globs. Rules and alerts are read from all matching files.
rule_files:
[ - <filepath_glob> ... ]
# Alerting specifies settings related to the Alertmanager.
alerting:
alert_relabel_configs:
[ - <relabel_config> ... ]
alertmanagers:
[ - <alertmanager_config> ... ]

In Grafana we can configure the alerts and channels using the UI easily and Test the connection.

Image for post
Image for post

The Startup

Medium's largest active publication, followed by +734K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store