Liveness and readiness probes for spring boot apps running in k8s

I have seen a lot of spring applications using /actuator/health endpoint as both k8s readiness and liveness probe. This may lead to unexpected behavior. Luckily spring announced liveness and readiness probes and made them available as part of 2.3.0 release recently that can fix it.

OK, so what is wrong with /actuator/health? According to the probes definitions:

• k8s is going to restart the pod if liveness probe failed
• and it is going to stop routing traffic if readiness probe failed

By default, spring boot auto-configuration includes application dependencies (such as jdbc datasource) in health endpoint components. In case of my application with postgres database that is:

>curl -XGET localhost:8080/actuator/health | jq
{
  "status": "UP",
  "components": {
    "db": {
      "status": "UP",
      "details": {
        "database": "PostgreSQL",
        "validationQuery": "isValid()"
      }
    },
    "diskSpace": {
      "status": "UP",
      "details": {
        "total": 226074402816,
        "free": 96203288576,
        "threshold": 10485760,
        "exists": true
      }
    }
  }
}

If the database is down it will return http 503 (service is down). When /actuator/health endpoint is used as liveness probe it will signal k8s to constantly restart application until the database is up again. Such behavior is not ideal as it:

• will mask the actual source of the problem: database
• will cause additional load on k8s by constantly restarting multiple services that depends on database, potentially stealing resources from healthy pods
• if there is automatic alerting in place it will complain about multiple services is down instead of just pointing to database

It is possible to fine-tune this behavior with probes.

First of all to start use them it is required to:

• upgrade to spring boot 2.3.0
• add dependency to org.springframework.boot:spring-boot-starter-actuator
• set property management.health.probes.enabled=true

This will expose 2 endpoints:

• /actuator/health/liveness
• /actuator/health/readiness

Out of the box they have exactly same behavior: both return status up if JVM is running and spring boot app is started. In this case application will not be restarted if or when db is down.

Next we might stop routing traffic to application if databases is unhealthy. This will allow opening circuit-breakers on client side and quickly fallback requests instead of waiting for query timeout. To do so set

management.endpoint.health.group.readiness.include=readinessState,db

to comma-separated list of HealthIndicator bean names without HealthIndicator suffix.

You could find sample project with both health and probe endpoints here.