Health Checks in SpringBoot

What do you know about Health Checks?” asked my manager in one of the daily standups. This question couldn’t have come at a worse time. After going through a dozen or so Sprints and spending the last few weeks getting the application ready for prime time, everyone in the team was ready for some R&R, certainly not in a mood to add some more code.

“We can specify a URL in the deployment descriptor that will be executed by the container repeatedly at a configured time interval. On error, a container will execute the health check URL some number times after which it reboots the service in the hope that it fixes the underlying issues,” I replied, cursing myself for not thinking about this aspect of the project.

“I talked to few other teams. They all have Health Checks in place. I think we should have it for our application as well,” said my manager.

Perfect another useless box to check before moving to production, I thought myself. “We can have a simple Ping Rest API and configure that to be the Health Check URL quickly,” I said. I patted myself in the back for coming up with that idea and felt like I saved the day.

“Health Checks are not just important, I would like them to be useful. I just don’t want it to be a bureaucratic box to check,” my manager said as if he read my mind.

“Even we developed a complex Health Check API, all that Container does is restart the application if the Health Check fails three times,” I said, standing my ground, getting the feeling that my dog is not hunting.

“Restarting the application instance is like surgery. We all go to annual physical to find out about our body. The doctor physically checks you, orders blood and other tests, compares reports, and, if there is a need, tweaks the medication. The idea here is to avoid or postpone an invasive action. I would like our Health Checks to generate a report about the application that helps me understand how it is doing,” quipped my manager.

“What exactly are we looking for the Health Check to do?” I mumbled, digesting the annual physical analogy and realizing that I am going to bring my bed to work knowing that a new WFH policy was in effect now.

“We had a lot of discussion about performance or our APIs. I would like to see how many times each of our APIs is called and how long it takes to serve those requests. It would be nice to see average times for the API calls. This will help to track the performance of our API’s and load distribution among the instances. I also would like to see if we are maxing out on connections in our DB connection pool. Obviously, we should make sure the external systems that we are dependent on are working,” answered manager, now in his avatar as Dr. Kevorkian.

“Sure. I will take a look at it” I said, knowing that my manager is clearly seeing what he wants out of Health Checks.

“I would like it to be done by the end of the week. I do not like to move go-live date,” said Dr. Kevorkian in his new Avatar as General Patten.

“I will try my best” is all that I can muster for a response.

Combination of SpringBoot HealthIndicator, HickariCP and Dropwizard Metrics will quickly and easily give us useful health information about the application.

Springboot HealthIndicator: HealthIndicator is an interface with just one method to implement. All you have to do is test series of conditions and return Health Object

1. @Override
2. public Health health() {
3. Health health;
4. if(dbConnectionTest())
5. health= Health.up().build();
6. else
7. health = Health.down().withDetail(“Health Issues”, -1).build();
8. reportMetrics();
9. return health;
10. }
11. protected void reportMetrics() {
12. Slf4jReporter reporter = Slf4jReporter.forRegistry(metricRegistry)
13. .outputTo(log)
14. .convertRatesTo(TimeUnit.SECONDS)
15. .convertDurationsTo(TimeUnit.MILLISECONDS)
16. .build();
17. reporter.report();
18. }

Configure in application.propertis file expose the HealthCheck url like this

1. management.endpoints.web.base-path=/
2. management.endpoints.web.path-mapping.health=health

Now your health check URL is ready and in working condition. Depending on what DB ORM tool you are using you can quickly write a DB connection check. Following code checks if DBConnection using a MyBatis Mapper.

1. @Mapper
2. @FunctionalInterface
3. public interface DbConnectionChecker {
4. @Select(“SELECT 1 FROM DUAL”)
5. Integer checkDbConnection();
6. }

HickariCP and DropWizard Metrics: HickariCP is the default connection pooling package for Springboot. HickariCP has a hook for DropWizard Metrics. DropWizard Metrics Once hooked to HickariCP, DropWizard generates very useful statistics about connection pool. A sample of connection pool stats collected by DropWizard-Hickari CP are shown at the end of this blog.

Configure HickariCP, hook it up with DropWizard and exposes the stats via url /metrics.

1. @Autowired
2. MetricRegistry metricRegistry;
3. @Bean
4. public DataSource primaryDataSource() {
5. Properties dsProps = new Properties();
6. dsProps.setProperty(“url”, dbUrl);
7. dsProps.setProperty(“user”, dbUserName);
8. dsProps.setProperty(“password”, dbPassowrd);
9. Properties configProps = new Properties();
10. configProps.setProperty(“connectionTestQuery”, “SELECT 1 FROM DUAL”);
11. configProps.setProperty(“driverClassName”, dbDriver);
12. configProps.setProperty(“jdbcUrl”, dbUrl);
13. HikariConfig hc = new HikariConfig(configProps);
14. hc.setDataSourceProperties(dsProps);
15. hc.setMetricRegistry(metricRegistry);
16. hc.setMaximumPoolSize(maxPoolSize);
17. hc.setPoolName(poolName);
18. return new HikariDataSource(hc);
19. }
20. @Bean
21. public MetricRegistry getMetricRegistery() {
22. return new MetricRegistry();
23. }
24. @Bean
25. public ServletRegistrationBean<MetricsServlet> servletRegistrationBean(){
26. return new ServletRegistrationBean<>(new MetricsServlet(),”/metrics/*”);
27. }
28. public class ReportsMetricsListener extends MetricsServlet.ContextListener{
29. @Autowired
30. MetricRegistry metricRegistry;
31. @Override
32. protected MetricRegistry getMetricRegistry() {
33. log.info(“Initializing Metrics Registry”);
34. return metricRegistry;
35. }
36. }

Performance Stats: DropWizard’s Meters can calculate execution time and report stats about any Methods. All you have to do is add some boiler plate code to the API you would like to measure. Meters measure execution time, number of times an API is called since last reboot and lot more. Using Spring Aspects annotations, we can insert boiler plate code without adding any code to the API’s. For this, lets develop an annotation called @meterIt shown below.

1. @Retention(RetentionPolicy.RUNTIME)
2. @Target(ElementType.METHOD)
3. public @interface MeterIt {}
4. @Aspect
5. @Component
6. public class MetricsExecutor {
7. @Autowired
8. MetricRegistry metricRegistry;
9. @Around(“@annotation(MeterIt)”)
10. public Object doMeterIt(ProceedingJoinPoint joinPoint) throws Throwable {
11. Timer.Context timerContext = metricRegistry.timer(joinPoint.getSignature().toShortString()).time();
12. Object object =joinPoint.proceed();
13. timerContext.stop();
14. return object;
15. }
16. }

Now you can gather all the statistics you would ever want about any API by just annotating it.

1. @MeterIt
2. @RequestMapping(value = “/nonDailyChecklistReport”, method = { RequestMethod.GET, RequestMethod.POST })
3. public ResponseEntity<byte[]> getNonDailyChecklistReport(
4. @RequestParam(value = “accountNo”, required = true) String accountNumber,
5. @RequestParam(value = “userId”, required = true) Integer userId,
6. @RequestParam(value = “offset”, required = true) Integer offset,
7. @RequestParam(value = “processingDate”, required = true) String processingDate,
8. @RequestParam(value = “langId”, required = true) Integer langId,
9. @RequestParam(value = “fundName”, required = true) String fundName,
10. @RequestParam(value = “userName”, required = true) String userName
11. ){
12. ByteArrayOutputStream baos = nonDailyChecklistReportService.generateCheckListReport(accountNumber, userId, offset, processingDate, langId, fundName, userName);
13. String reportName = “Non-Daily Checklist Report.pdf”;
14. return createResponse(baos, reportName, “pdf”);
15. }

An Example of Connection Pool stats generated by HickariCP and DropWizard …

1. “gauges”: {
2. “ReportsApiPool.pool.ActiveConnections”: {
3. “value”: 0
4. },
5. “ReportsApiPool.pool.IdleConnections”: {
6. “value”: 5
7. },
8. “ReportsApiPool.pool.MaxConnections”: {
9. “value”: 5
10. },
11. “ReportsApiPool.pool.MinConnections”: {
12. “value”: 5
13. },
14. “ReportsApiPool.pool.PendingConnections”: {
15. “value”: 0
16. },
17. “ReportsApiPool.pool.TotalConnections”: {
18. “value”: 5
19. }
20. },
21. “counters”: {},
22. “histograms”: {
23. “ReportsApiPool.pool.ConnectionCreation”: {
24. “count”: 19,
25. “max”: 363,
26. “mean”: 324.46097589119296,
27. “min”: 231,
28. “p50”: 328,
29. “p75”: 333,
30. “p95”: 363,
31. “p98”: 363,
32. “p99”: 363,
33. “p999”: 363,
34. “stddev”: 19.155332533559314
35. },
36. “ReportsApiPool.pool.Usage”: {
37. “count”: 348,
38. “max”: 105,
39. “mean”: 22.030874720896463,
40. “min”: 0,
41. “p50”: 33,
42. “p75”: 33,
43. “p95”: 33,
44. “p98”: 33,
45. “p99”: 33,
46. “p999”: 41,
47. “stddev”: 15.584664000769294
48. }
49. },