Custom Health Indicators in Spring Boot

Introduction

Spring Boot provides Health Indicators to monitor application health and database health with the help of Spring Boot Actuator. Spring Boot Actuator comes with various Health Indicators for most of the relational databases and non-relational databases like MongoDB, Redis, ElasticSearch etc. Spring Boot Actuator also provides health indicators for RabbitMQ, IBM MQ out of the box.

Why to go for Custom Health Checks

If we want to monitor health status of external services our application connects to, Spring Boot doesn’t provide any health indicators for this scenario out of the box, we need to write custom health indicators. Also Spring Boot doesn’t provide any kafka health indicator out of the box as of today, so in this scenario we need to go for custom health indicator.

Implementation

project structure

Main class

package com.lights5.services.orders.api;

import com.lights5.services.orders.api.config.EndpointsRegistry;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class OrdersApiApplication {

 @Autowired
 private EndpointsRegistry endpointsRegistry;

 public static void main(String[] args) {
  SpringApplication.run(OrdersApiApplication.class, args);
 }
}

application.yml file

spring:
  application:
    name: orders-api
  kafka:
    producer:
      bootstrap-servers: localhost:9092
  data:
    mongodb:
      host: localhost
      port: 27017
      database: orders_db
      username: ${username}
      password: ${pwd}
      authentication-database: orders_db

app:
  client:
    payments-service:
      host: https://example.com
      paths:
        health-check: /mock-service/health
        intiate-payment: /payments

ApplicationConfig

package com.lights5.services.orders.api.config;

import org.apache.kafka.clients.admin.AdminClient;
import org.springframework.boot.actuate.data.mongo.MongoHealthIndicator;
import org.springframework.boot.autoconfigure.kafka.KafkaProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.mongodb.core.MongoTemplate;

import java.util.Properties;

@Configuration
public class AppConfig {

    @Bean
    MongoHealthIndicator mongoHealthIndicator(MongoTemplate mongoTemplate) {

        return new CustomMongoHealthIndicator(mongoTemplate);
    }

    @Bean
    AdminClient kafkaAdminClient(KafkaProperties kafkaProperties) {

        Properties properties = new Properties();
        properties.put("bootstrap.servers", kafkaProperties.getBootstrapServers());
        properties.put("request.timeout.ms", 3000);
        properties.put("connections.max.idle.ms", 5000);
        return AdminClient.create(properties);
    }
}

EndpointsRegistry

This class registers all external clients that application connects to.

package com.lights5.services.orders.api.config;

import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;

import java.util.Map;

@Getter
@Setter
@Configuration
@ConfigurationProperties(prefix = "app")
public class EndpointsRegistry {

    private Map<String, ServiceEndpoint> client;

    @Getter
    @Setter
    static class ServiceEndpoint {

        private String host;
        private Map<String, String> paths;
    }

    public String getHealthCheckURL(String serviceName) {
        ServiceEndpoint endpoint = this.client.get(serviceName);
        if (endpoint != null) {
            String healthCheckPath = endpoint.getPaths().get("health-check");
            String host = endpoint.getHost();

            return host + healthCheckPath;
        }
        return null;
    }
}

PaymentServiceMonitor (External Client)

This class overrides the default behaviour of AbstractHealthIndicator class. It executes GET request on health check url of external client and updates the status accordingly.

package com.lights5.services.orders.api.config;

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
import org.apache.hc.client5.http.impl.classic.HttpClientBuilder;
import org.springframework.boot.actuate.health.AbstractHealthIndicator;
import org.springframework.boot.actuate.health.Health;
import org.springframework.stereotype.Component;

import java.io.IOException;

@Slf4j
@Component
@RequiredArgsConstructor
public class PaymentServiceHealthIndicator extends AbstractHealthIndicator {

    private final EndpointsRegistry endpointsRegistry;

    @Override
    protected void doHealthCheck(Health.Builder builder) {
        try {
            String healthCheckURL = endpointsRegistry.getHealthCheckURL("payment-service");
            CloseableHttpClient httpClient = HttpClientBuilder.create().build();

            CloseableHttpResponse response = httpClient.execute(new HttpGet(healthCheckURL));
            if (response.getCode() >= 500) {
                log.error("Payment Service is down");
            }
            else {
                builder.up();
            }
        } catch (IOException ex) {
            builder.down();
        }
    }
}

HostAvailability Class

This class is used to verify whether application is able to connect to given host and port number.

package com.lights5.services.orders.api.service;

import lombok.extern.slf4j.Slf4j;

import java.io.IOException;
import java.net.*;

@Slf4j
public class HostAvailability {

    private HostAvailability() {
        // to prevent instantiation from other classes.
    }

    public static boolean isAvailable(String hostName, int port) {
        SocketAddress socketAddress = new InetSocketAddress(hostName, port);

        try (Socket socket = new Socket()) {

            socket.connect(socketAddress, 5000);
        } catch (IOException e) {

            log.error("Application Health Check Failed due to service unavailability {}", e.getMessage());
            return false;
        }
        return true;
    }
}

CustomMongoHealthIndicator

This class extends the MongoHealthIndicator (provided by spring boot) and oveerides the behaviour of health check method. First we are verifying whether app is able to connect to host and port and then we are executing small command on database to check database connectivity. If we execute command directly without verifying server connectivity, it takes more time if server is not reachable from application. That’s why we are verifying server connectivity first.

package com.lights5.services.orders.api.config;

import com.lights5.services.orders.api.service.HostAvailability;
import lombok.extern.slf4j.Slf4j;
import org.bson.Document;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.actuate.data.mongo.MongoHealthIndicator;
import org.springframework.boot.actuate.health.Health;
import org.springframework.data.mongodb.core.MongoTemplate;

@Slf4j
public class CustomMongoHealthIndicator extends MongoHealthIndicator {

    private final MongoTemplate mongoTemplate;

    @Value("${spring.data.mongodb.host}")
    private String mongodbHost;

    @Value("${spring.data.mongodb.port}")
    private int port;

    public CustomMongoHealthIndicator(MongoTemplate mongoTemplate) {
        super(mongoTemplate);
        this.mongoTemplate = mongoTemplate;
    }

    public void doHealthCheck(Health.Builder builder) throws Exception {
        boolean isServerAvailable = HostAvailability.isAvailable(mongodbHost, port);
        if (isServerAvailable) {
            Document result = mongoTemplate.executeCommand("{ isMaster: 1 }");
            builder.up().withDetail("maxWireVersion", result.getInteger("maxWireVersion"));
        }
        else {
            log.error("MongoDB Server is down.");
            builder.down();
        }
    }
}

KafkaHealthIndicator

This class provides implementation to verify kafka cluster health. It uses KafkaAdminClient to describe the cluster, if get the response then cluster is up else cluster is down.

Note: Doesn’t work if producer is configured to support transactions.

package com.lights5.services.orders.api.config;

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.common.Node;
import org.springframework.boot.actuate.health.AbstractHealthIndicator;
import org.springframework.boot.actuate.health.Health;
import org.springframework.stereotype.Component;

import java.util.Collection;
import java.util.concurrent.TimeUnit;

@Slf4j
@Component
@RequiredArgsConstructor
public class KafkaHealthIndicator extends AbstractHealthIndicator {

    private final AdminClient kafkaAdminClient;

    @Override
    protected void doHealthCheck(Health.Builder builder) throws Exception {

        Collection<Node> nodes = kafkaAdminClient.describeCluster()
                .nodes().get(3, TimeUnit.SECONDS);

        if (!nodes.isEmpty()) {
            log.error("Kafka Server is up with nodes {}", nodes.size());
            builder.up();
        }
        else {
            log.error("Kafka Server is down");
            builder.down();
        }
    }
}

Working

If any of these HealthIndicators (MongoDB, Kafka, Payments Service) is down, then health check url of application returns status as

{
  "status": "DOWN"
}

If all these dependent services are up, then the status will be returned as

{
  "status": "UP"
}

Endpoint to verify app health: http://localhost:8080/actuator/health

Conclusion

Custom Health Checks are important as it provides greater visibility of external services availability. We can take action immediately there by decreasing application errors.