Application Monitoring with Prometheus and Grafana (Part 3)

In this series, we have created the client applications with REST APIs and used Spring Boot Actuators to monitor the health of the applications, metrics for the REST APIs, Database Connections & Status, Tomcat and Hikari Connections, Cache Management, Scheduled Tasks, Environment of an application and much more. To visualize and monitor the applications we need UI, to achieve this we have created Spring Boot Admin which acts as a server application and uses JSON information from Actuators and provides visualization in form of graphs and tables.

• Application Monitoring using Spring Boot Actuators (Part 1)
• Application Monitoring using Spring Boot Admin (Part 2)
• Application Monitoring using Prometheus and Grafana

Prometheus

Prometheus is an open-source system that provides a more flexible and great visualized information, alert mechanism, rules, powerful queries, storage, etc. Most Prometheus components are written in Go, making them easy to build and deploy as static binaries.

Architecture

Architecture by Prometheus

Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be used to visualize the collected data.

Configurations

Let’s download the Prometheus with OS like windows and architecture as amd64. Here I have used the prometheus-2.16.0.windows-amd64. Extract it at an appropriate place and you can see prometheus.yml where we can customize the Prometheus with scarp intervals, alert managers and rules, etc. We can also specify which configuration file to load, using --config.file flag.

Before going ahead, make sure we have done the required configurations in our application.

• Include spring-boot-starter-actuator, micrometer-registry-Prometheus in our pom.xml
• /prometheus endpoint is exposed in application-properties
• We can use enable and expose configurations with Spring Security to prevent unauthorized access to sensitive information.
• metrics_path and scrape_interval is configured correctly in prometheus.yml

pom.xml
<dependency>
   <groupId>io.micrometer</groupId>
   <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
application-properties
#Include end-points
management.endpoint.metrics.enabled = true
management.endpoints.web.exposure.include = *
management.endpoint.metrics.show-details = always
management.endpoint.health.show-details=always

Grafana

Grafana is the most popular open-source project that allows you to query, visualize and alert on metrics and logs no matter where they are stored. It uses time-series databases like Graphite, Prometheus, Elasticsearch, OpenTSDB and InfluxDB to query. We can also integrate Grafana with Splunk, AppDynamics, DataDog, and support Custom Auth (LDAP, OAuth, and more), SSL and Custom Domains (enterprise version). For visualization, we can use ready to use dashboards provided by it. You can download the latest version from here.

Add Prometheus as Datasource

Before going ahead, we have to add a data source in Grafana and can create the dashboard for application monitoring and visualization.

Add DataSource

Let’s Run & Monitor

• Prometheus can be accessed on localhost:9090
• Grafana can be accessed on localhost:3000

Choose a metric, execute & analyze

Prometheus: localhost:9090/graph

Core Statistics

Grafan: localhost:3000

REST APIs

Grafan: localhost:3000

JVM Statistics

Grafan: localhost:3000

Hikari Connections

Grafan: localhost:3000

Logs

Grafan: localhost:3000

Rules & Alert Mechanism

In Prometheus and Grafana, we can configure the alert mechanism and rules according to the requirements and take immediate actions based on health, load, and connections of the instances.

Define Rules and Alerts in Prometheus

# Rule files specifies a list of globs. Rules and alerts are read from all matching files.
rule_files:
  [ - <filepath_glob> ... ]
# Alerting specifies settings related to the Alertmanager.
alerting:
  alert_relabel_configs:
    [ - <relabel_config> ... ]
  alertmanagers:
    [ - <alertmanager_config> ... ]

Configure Alerts and Notification Channels in Grafana

In Grafana we can configure the alerts and channels using the UI easily and Test the connection.