Monitoring and Observability with Spring Boot 3
In this blog post, I’ll start by providing an overview of monitoring, observability with their respective tools. Then, I will explore monitoring Spring Boot applications using Actuator and Micrometer Observability API. I will cover the setup process and highlight the benefits of using these tools.
Monitoring and observability are crucial for effectively managing large-scale production applications. Monitoring allows for data collection regarding the application’s performance and health, while observability enables understanding and troubleshooting when issues arise.
Monitoring involves collecting and analyzing metrics and logs to provide a comprehensive understanding of the system’s behavior. This involves gathering metrics, logs, traces, and distributed tracing.
In essence, monitoring focuses on ensuring the system’s health (“Are we okay?”), whereas observability is about understanding issues and deriving insights from them (“Why are we not okay, and what can we learn from it?”).
There are various tools available for monitoring and observability. Here are some popular ones:
1. Prometheus — an open-source systems monitoring and alerting toolkit originally built at SoundCloud.
2. Grafana — a visualization and analytics data from multiple sources, including Prometheus
3. Splunk — It’s used for searching, monitoring, and analyzing machine-generated data, including logs, events, and other types of data, in real-time.
4. Loki — like Prometheus, but for logs.
5. OpenTelemetry —It’s the observability framework which provides a standard way to collect observability data and export these data(metrics, traces, and logs) from our applications.
6. Zipkin — a distributed tracing system. It helps gather timing data needed to troubleshoot latency problems in service architectures. Features include both the collection and lookup of this data.
First, we need to create a spring boot application: https://start.spring.io/ or you can find the full example in my github repository: https://github.com/minarashidi/transfer-service/tree/main
In this example, I have used prometheus as the monitoring backend and Grafana to create dashboards to visualize and analyze data. The metrics of the service is exposed through http and prometheus scrapes/collects metrics at that endpoint at regular intervals.
Spring Boot Actuator
It provides a set of endpoints that can be used to monitor and manage your application. These endpoints can be used to collect data on a variety of metrics, such as CPU usage, memory usage, request rates, and error rates.
The spring-boot-actuator module provides all of Spring Boot’s production-ready features, and it’s configured to expose metrics on /prometheus.
All metrics are tagged with the application name in the spring configuration; e.g. application=transfer-service
To enable the features we need to add below dependencies:
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>Actuator includes a variety of built-in endpoints, allowing you to add your own. Each of these endpoints is prefixed with /actuator. For instance, the health endpoint offers basic application health information and is mapped to /actuator/health by default.
And actuator endpoint at /actuator/prometheus which we can change it to /metrics
path-mapping:
prometheus: "metrics"To enable the endpoints, add this config to the application’s yaml file.
management:
server:
port: 9101
endpoints:
web:
exposure:
include: health, prometheus
base-path: "/"
path-mapping:
prometheus: "metrics"
metrics:
tags:
application: transfer-servicePrometheus
Now we can configure Prometheus to scrape or poll individual application instances for metrics, and as we configured Spring Boot provides an actuator endpoint at /metrics to present a Prometheus scrape with the appropriate format.
we add the scrape_config inprometheus.yml
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
static_configs:
- targets: [ 'localhost:9090' ]
- job_name: 'transfer-service'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 2s
metrics_path: /metrics
static_configs:
- targets: [ 'host.docker.internal:9101' ]Grafana
Ensure to add Prometheus as a Data Source in Grafana.
You can find examples on how to create dashboards here: https://github.com/minarashidi/transfer-service/tree/main/scripts/cfg/grafana
Run monitoring components using docker-compose
We need to create a docker-compose file to include the whole observability infrastructure
version: "3"
services:
prometheus:
image: prom/prometheus
container_name: prometheus
volumes:
- ./cfg/prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- 9090:9090
grafana:
image: grafana/grafana
container_name: grafana
restart: always
ports:
- 3003:3003
volumes:
- ./cfg/grafana/provisioning:/etc/grafana/provisioning
- ./cfg/grafana/grafana.ini:/etc/grafana/grafana.ini
env_file:
- ./cfg/grafana/grafana.env
zipkin:
image: openzipkin/zipkin
container_name: zipkin
ports:
- "9411:9411" # Expose the Zipkin UI portFollow these steps to run the infrastructure to start up the observability stack:
docker compose up
mvn spring-boot:run
Access the following endpoints once your application is running:
- To access Prometheus: http://localhost:9090
- To access Grafana Dashboard: http://localhost:3003
- To access Zipkin UI: http://localhost:9411
Observability
It is the ability to observe the internal state of a running system from the outside.
It consists of the three pillars logging, metrics and traces.
Micrometer Observability API is a lightweight library for instrumenting your application with metrics. It provides a simple API for tracking metrics, and it integrates with a variety of monitoring and observability tools.
For metrics and traces, Spring Boot we uses Micrometer Observation.
To create your own observations (which will lead to metrics and traces), you can inject an ObservationRegistry.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>import io.micrometer.observation.ObservationRegistry;
import io.micrometer.observation.aop.ObservedAspect;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class ApplicationConfig {
// Monitoring beans
@Bean
ObservationRegistry observationRegistry() {
return ObservationRegistry.create();
}
// To have the @Observed support we need to register this aspect
@Bean
ObservedAspect observedAspect(ObservationRegistry observationRegistry) {
return new ObservedAspect(observationRegistry);
}
}public interface MetricsService {
void countCreatedDeposits(Deposit deposit);
}
package com.mina.transferservice.metric;
import com.mina.transferservice.domain.Deposit;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.observation.Observation;
import io.micrometer.observation.ObservationRegistry;
import lombok.AllArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
@AllArgsConstructor
@Slf4j
@Component
public class DepositMetricsService implements MetricsService {
public static final String CREATED_TRANSFERS = "created_deposits";
public static final String KAFKA_NOTIFICATIONS = "kafka_notifications";
private ObservationRegistry observationRegistry;
private MeterRegistry meterRegistry;
@Override
public void countCreatedDeposits(Deposit deposit) {
Observation.createNotStarted(CREATED_DEPOSITS, observationRegistry)
.lowCardinalityKeyValue("request-uid", transfer.getRequestUid())
.observe(() -> {
// Execute business logic here
log.debug("Counting created deposits")
});
}
...
}Low cardinality tags will be added to metrics and traces, while high cardinality tags will only be added to traces.
And then in your service you can call it like this:
import com.mina.transferservice.domain.Deposit;
import com.mina.transferservice.metric.MetricsService;
import com.mina.transferservice.repository.TransferRepository;
import lombok.AllArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Slf4j
@Service
@AllArgsConstructor
@Transactional(readOnly = true)
public class DepositService {
private final DepositRepository depositRepository;
private MetricsService serviceMetrics;
@Transactional
public Deposit add(Deposit deposit) {
var depositOptional = depositRepository.findByRequestUid(deposit.getRequestUid());
return transferOptional.orElseGet(() -> {
log.debug("New deposit request, requestUid: {}", deposit.getRequestUid());
var savedDeposit = depositRepository.save(deposit);
serviceMetrics.countCreatedDeposits(savedDeposit);
return savedDeposit;
});
}
Using Annotations With @Observed
We need to add ObservedAspec bean to use the @Observed annotation to create observations. You can put that annotation either on a method to observe it or a class to observe all methods in it.
When we mark a method with@Observed, Micrometer will automatically collect metrics about it on a variety of things, such as:
- Method execution time: The time it takes for a method to execute.
- Field values: The values of fields.
- Counter: A counter that increments each time a certain event occurs.
- Gauge: A gauge that measures the current value of a certain metric.
- Timer: A timer that measures the duration of a certain operation.
@Override
@Observed(name = DepositMetricsService.KAFKA_NOTIFICATIONS)
public void sendOrderConfirmation(DepositOrder order) {
// send notification message async using Java 21 virtual threads
}Logs
I will first talk about logging, since for tracing we need to include the current trace and span id in the logs:
For logging, using Logback and configuring it for different log formats based on the logback_appenders property to use:
- LogstashEncoder for JSON formatting
- Custom pattern for non-JSON formatting
<configuration>
<!-- Conditionally choose the appender based on the logback_appenders property -->
<if condition='property("logback_appenders").contains("json")'>
<then>
<!-- Use LogstashEncoder for JSON formatting -->
<appender class="ch.qos.logback.core.ConsoleAppender" name="logAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
</appender>
</then>
<else>
<!-- Use a custom pattern for non-JSON formatting -->
<appender class="ch.qos.logback.core.ConsoleAppender" name="logAppender">
<encoder>
<charset>UTF-8</charset>
<pattern>%green(%d{ISO8601}) %highlight(%-5level) [%blue(%t)] [%X{traceId:-}] [%X{spanId:-}] %yellow([%X{order:-}]) %logger -
%msg%n%throwable
</pattern>
</encoder>
</appender>
</else>
</if>
<root level="INFO">
<appender-ref ref="logAppender"/>
</root>
<logger name="com.mina.transferservice" level="DEBUG"/>
</configuration>
<!--Format log messages to include:
The date and time in ISO 8601 format
The log level, displayed in a highlighted color.
The thread name, displayed in blue.
The trace ID, displayed in blue.
The span ID, displayed in blue.
The order of the log event, displayed in yellow.
The logger name.
The log messages.
A newline character.
The stack trace of any exception that was thrown.
-->Tracing
Instead of using Sleuth, which is no longer used in Spring Boot 3, we need to use micrometer-tracing, which provides a simple facade for the most popular tracer libraries, and letting us instrument our application code without vendor lock-in. https://docs.spring.io/spring-boot/docs/current/reference/html/actuator.html#actuator.micrometer-tracing
Now we have to add the following dependencies:
<!--OpenTelemetry Tracer: Bridges the Micrometer Observation API to OpenTelemetry-->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<!--Reporter/Zipkin with OpenTelemetry: Reports traces to Zipkin-->
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-zipkin</artifactId>
</dependency>Add the following application properties file:
management:
tracing:
sampling:
probability: 1.0By default, Spring Boot samples only 10% of requests to prevent overwhelming the trace backend. This property switches it to 100% so that every request is sent to the trace backend.
To collect and visualize the traces, we need a running trace backend which we can use Zipkin as our trace backend here.
Behind the scenes, an observation has been created for the HTTP request, which in turn gets bridged to OpenTelemetry, which reports a new trace to Zipkin.
Now if we open the Zipkin UI at localhost:9411 and press the "Run Query" button to list all collected traces. You should see one trace. Press the "Show" button to see the details of that trace.
OpenTelemetry
OpenTelemetry is the observability framework which provides a standard way to collect observability data and export these data(metrics, traces, and logs) from our applications. It is vendor-neutral and supports a variety of backends, including Zipkin. It means we’re not locked into a specific tracing or monitoring tool. We can switch or use multiple backends as needed.
I hope this post helped you to gain better understanding on how we can add monitoring and observability in our spring boot applications and you can find more details in my github repository: https://github.com/minarashidi/transfer-service
