Monitoring Spring Boot Microservices

Build comprehensive monitoring capabilities for Spring Boot Microservice using Spring Boot Actuator, Micrometer and Prometheus.

Introduction

Observability, which is comprised of monitoring, logging, tracing, and alerting aspects, is an important architectural concern when using microservices and event-driven architecture (EDA) styles, primarily because:

• A large number of deployments require automation and centralization of monitoring/observability
• The asynchronous and distributed nature of the architecture results in difficulties related to correlating metrics produced from multiple components

Addressing this architectural concern provides simplified management and quick turn-around time for resolving runtime issues. It also provides insights that can help in making informed architectural, design, deployment, and infrastructure decisions to improve non-functional characteristics of the platform. Additionally, useful business/operations insights can be obtained by engineering emission, collection, and visualization of custom metrics.

Monitoring is mainly comprised of the following four sets of activities:

• Instrumentation of the application(s) — Instrumenting the application to emit the metrics that are of importance to the application monitoring and maintenance teams, as well as for the business users. There are many non-intrusive ways for emitting metrics, the most popular ones being “byte-code instrumentation,” “aspect-oriented programming,” and “JMX.”
• Metrics collection — Collecting metrics from the applications and persisting them in a repository/data-store. The repository then provides a way to query and aggregate data for visualization. Some of the popular collectors are Prometheus, StatsD, and DataDaog. Most of the metrics collection tools are time-series repositories and provide advanced querying capability.
• Metrics visualization — Visualization tools query the metrics repository to build views and dashboards for end-user consumption. They provide rich user interface to perform various kinds of operations on the metrics, such as aggregation, drill-down, and so on.
• Alerts and Notifications — When metrics breach defined thresholds (for instance CPU is more than 80% for more than 10 minutes), human intervention might be required. For that, alerting and notifications are important. Most visualization tools provide alerting and notification ability.

Spring Boot Actuator

Actuator is a set of features that help us monitor and manage our application when it moves away from local development environment and onto a test, staging or production environment. It helps expose operational information about the running application — health, metrics, audit entries, scheduled task, env settings, etc. We can query the information via either several HTTP endpoints or JMX beans.

To help with the monitoring and management of a microservice, enable the Spring Boot Actuator by adding spring-boot-starter-actuator as a dependency.
Dependency addition is as follows:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Actuator also creates an endpoint for metrics. By default, it is /actuator/metrics. It needs to be exposed through Spring configuration. The following is a sample configuration:

management:
  endpoints:
    web:
      exposure:
        include: ["health", "info", "metrics", "prometheus", "bindings", "beans", "env", "loggers", "streamsbindings", "mappings"]

To check, let’s navigate our browser to http://localhost:8080/actuator:

Micrometer

Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing us to instrument our JVM-based application code without vendor lock-in.

They quite correctly describe themselves as:

Think SLF4J, but for metrics.

Just as a refresher, SLF4J is a logging facade for other Java logging frameworks. SLF4J itself does not have any logging implementation. The idea is that you write code using SLF4J API’s and the real implementation of it comes from the framework you choose. It could be any of the popular frameworks like log4j, logback, etc.

Similarly, Micrometer automatically exposes /actuator/metrics data into something our monitoring system can understand. All we need to do is include that vendor-specific micrometer dependency in our application.

Integration with Prometheus

Since Prometheus uses polls to collect metrics, it is relatively simple two-step process to integrate Prometheus and Micrometer.

1. Add the micrometer-registry-prometheus registry.

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

2. Declare a bean of type MeterRegistryCustomizer<PrometheusMeterRegistry>.

@Configuration
public class MicroSvcMeterRegistryConfig {
    @Value("${spring.application.name}")
    String appName;

    @Value("${env}")
    String environment;

    @Value("${instanceId}")
    String instanceId;

    @Bean
    MeterRegistryCustomizer<PrometheusMeterRegistry> configureMetricsRegistry()
    {
        return registry -> registry.config().commonTags("appName", appName, "env", environment, "instanceId", instanceId);
    }

This is an optional step. However, it is recommended, as it provides a mechanism to customize the MeterRegistry. This is useful for declaring common tags (dimensions) for the metrics data that would be collected by Micrometer. This helps in metrics drill-down. It is especially useful when there are a lot of microservices and/or multiple instances of each microservice. Typical common tags could be applicationName, instanceName, and environment. This would allow you to build aggregated visualizations across instances and applications as well as be able to drill down to a particular instance/application/environment.

Rebuild and start the application and navigate our browser to http://localhost:8080/actuator:

This will generate a new endpoint — /actuator/prometheus. Opening it, we will see data formatted specific for Prometheus:

Instrumentation of application-level metrics

A microservice would typically have Controller, Service, DAO, and Integration layers.

The quickest and easiest way to instrument REST controllers is to use the @Timed annotation on the controller or on individual methods of the controller. @Timed automatically adds these tags to the timer: exception, method, outcome, status, uri. It is also possible to supply additional tags to the @Timed annotation.

For Service, DAO, and Integration layers, developers create custom beans annotated with @Service or @Component annotations. Metrics related to latency, throughput, and exceptions can provide vital insights. These can be easily gathered using Micrometer’s Timer and Counter metrics. However, the code needs to be instrumented for applying these metrics. A common reusable class that instruments services and components can be created using spring-aop, which would be reusable across all microservices. Using @Around and @AfterThrowing advice metrics can be generated without adding any code to the service/component classes and methods.

Take a look at the following sample annotation:

@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface MonitoredService {
}

The following code shows a sample reusable aspect that can instrument the Service classes:

@Configuration
@EnableAspectJAutoProxy
@Aspect
public class MonitoringAOPConfig {
    @Autowired
    MeterRegistry registry;
    @Pointcut("@target(com.ibm.dip.microsvcengineering.framework.monitoring.MonitoredService) && within(com.ibm.dip..*)")
    public void servicePointcut() {
    }
    @Around("servicePointcut()")
    public Object serviceResponseTimeAdvice(ProceedingJoinPoint pjp) throws Throwable {
        return monitorResponseTime(pjp, TAG_VALUE_SERVICE_TYPE);
    }
    @AfterThrowing(pointcut = "servicePointcut()", throwing = "ex")
    public void serviceExceptionMonitoringAdvice(JoinPoint joinPoint, Exception ex)
    {
        monitorException(joinPoint, ex, TAG_VALUE_SERVICE_TYPE);
    }
    private Object monitorResponseTime(ProceedingJoinPoint pjp, String type) throws Throwable {
        long start = System.currentTimeMillis();
        Object obj = pjp.proceed();
        pjp.getStaticPart();
        long end = System.currentTimeMillis();
        String serviceClass = getClassName(pjp.getThis().getClass().getName());
        String methodName = pjp.getSignature().getName();
        Timer timer = registry.timer(METER_COMPONENT_TIMER,
                TAG_COMPONENTCLASS, serviceClass, TAG_METHODNAME, methodName, TAG_OUTCOME, SUCCESS, TAG_TYPE, type);
        timer.record((end - start), TimeUnit.MILLISECONDS);
        Counter successCounter = registry.counter(METER_COMPONENT_COUNTER,
                TAG_COMPONENTCLASS, serviceClass, TAG_METHODNAME, methodName, TAG_OUTCOME, SUCCESS, TAG_TYPE, type);
        successCounter.increment();
        return obj;
    }
    private void monitorException(JoinPoint joinPoint, Exception ex, String type)
    {
        String serviceClass = getClassName(joinPoint.getThis().getClass().getName());
        String methodName = joinPoint.getSignature().getName();
        Counter failureCounter = registry.counter(METER_COMPONENT_EXCEPTION_COUNTER, TAG_EXCEPTIONCLASS,
                ex.getClass().getName(), TAG_COMPONENTCLASS, serviceClass, TAG_METHODNAME, methodName, TAG_OUTCOME, ERROR,
                TAG_TYPE, type);
        failureCounter.increment();
    }
}

A sample instrumented Service class will have the following annotations on it. Automatically, all the methods in this Service class will become candidates for applying the serviceResponseTimeAdvice and serviceExceptionMonitoringAdvice.

@Service
@MonitoredService
public class SampleService {
   ...
}

Visualization of metrics

Here we will use Prometheus as our visualization tool. Prometheus is a time-series database that stores our metric data by pulling it periodically over HTTP. The intervals between pulls can be configured and we have to provide the URL to pull from. It has a simple user interface where we can visualize/query on all of the collected metrics.

Let’s configure Prometheus, and more precisely the scrape interval, the targets, etc. To do that, we’ll be using the prometheus.yml file:

global:
  scrape_interval: 10s
scrape_configs:
  - job_name: 'spring_micrometer'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.2.8:8080']

Since we are using Docker to run Prometheus, it will be running in a Docker network that won’t understand localhost/120.0.01. So instead of using locahost:8080, 192.168.2.8:8080 is used where 192.168.2.8 is my PC IP at the moment.

Now, we can run Prometheus using the Docker command:

$ docker run -d -p 9090:9090 -v <path-to-prometheus.yml>:/etc/prometheus/prometheus.yml prom/prometheus

To see Prometheus dashboard, navigate to http://localhost:9090

To check if Prometheus is actually listening to the Spring app, we can go to the /targets endpoint:

Let’s go back to the home page and select a metric from the list and click Execute:

We can also use Grafan for metrics visualization. Grafana offers a rich UI where we can build up custom graphs quickly and create a dashboard out of many graphs in no time.

Conclusion

Monitoring an application’s health and metrics helps us manage it better, notice unoptimized behavior, and better understand its performance. This especially holds true when we’re developing a system with many microservices, where monitoring each service can prove to be crucial when it comes to maintaining our system.

Based on this information, we can draw conclusions and decide which microservice needs to scale if further performance improvements can’t be achieved with the current setup.

In this article, we used Micrometer to reformat the metrics data provided by Spring Boot Actuator and expose it in a new endpoint. This data was then regularly pulled and stored by Prometheus, which is a time-series database. Ultimately, we’ve used Prometheus to visualize this information with a user-friendly dashboard.

References

[1] https://developer.ibm.com/languages/java/tutorials/monitor-spring-boot-microservices/