Health Check Rack Middleware

Let Us Support Health Check Endpoint for All Services Using Rack Middleware

Al-Waleed Shihadeh
Jan 19 Β· 7 min read
Photo by Hush Naidoo on Unsplash

Implementing health check endpoints for software applications is a pattern in the software development process and it is one of the best practices. Health check endpoints can be used for several purposes, below is a list of some of the use cases for health check endpoints:

  • Monitoring systems: Health check endpoints can be used as a source of data for monitoring systems that keep track of service status, based on the responses of the endpoints actions can be taken. For instance, send an SMS notification to the emergency manager if one of the endpoints responds with a 503 error code.
  • Load Balancers: Load balancers or reverse proxies can utilize the health check endpoints to include or excludes nodes or servers form the service pool. For instance, In the case of a load balancer that balances the load between a set of nodes which is running a given service. If the service is down on one of the nodes, the load balancer will automatically stop forwarding requests to that nodes and as a result, the end-users will not be affected.
  • Container clusters: Container clustering solutions such as swarm or kubernetes can also get some benefits from services health check endpoints. For Instance, swarm uses service health checks to decide healthy services from unhealthy ones and therefore restart the unhealthy one or take other actions based on the service configurations.
  • Software Deployments: Health check endpoints can be also used to automate the deployment process and achieve zero-downtime deployment by checking for service dependencies and wait for them to be ready before the deployment or rolling back to an older healthy version of the service in case the new version is not healthy. A good example here is rolling out new updates for docker service, Instead of just replacing old containers with the new ones in one step, we can start the new containers, checks the health of the new containers and only if they are healthy we proceed with the replacement process.
  • Installation verifications: Health check endpoints are also a great help in investigation use cases, especially if these endpoints not only reports the health of the service itself but also the health of its dependencies. For instance, if a service has a dependency on Redis , MySQL, and RabbitMQ and it provides health endpoints that report the status of the connections between the service and these dependencies.

Due to the above use cases, there is a need to implement the health check endpoints in almost every service. The implementation of the health check endpoints is very simple and can be just a static request that returns the status of the services, However, implementing the health check in several services can introduce some maintainability concerns due to the fact that we are duplicating the codes in all the supported services and if we decided to update the health endpoints (for some reason πŸ˜ƒ) we need to do the same change on all the services. Or in the case of new services, we need to spend some time implementing the health checks and testing them for the new service. In addition, especially in the case of multiple teams working on implementing the health checks, we may end up with inconsistent health check endpoints across the supported services.

For the above reason, It makes a lot of sense to try to implement the health checks endpoint into a software library and integrate the library with all supported services. In the case of Rails/Rack software applications, this task can be easily achieved by building a rack middleware. To illustrate more, the web requests are handled first by a web server such as puma or unicorn, these servers proxies the requests to the applications server, The request will be processed by the integrated middleware and then it will be handled by the web framework such Sinatra or Rails. This feature helps in providing fast responses for the health checks since it does not need to be processed by the web framework and can be served to the client by the middleware layer.

In this post, I will describe my attempt to build a rack middleware that provides health check endpoints for ruby applications that uses rack.

Middleware Features

Below is the list of the requirements of the health check middleware that I needed for my rails applications :

  • Support multiple health check endpoint that covers the service health and the dependencies of the service.
  • The middleware is configurable.
  • The middleware responses should be JSON responses.

To cover the first item in the list above, I decided to support the following health check endpoints

  • \health : a Static health check endpoint that can be used to check the status of the application server if it is running or not. The response of this endpoint should be the below JSON
{
"data": {
"type": "checker",
"id": "1",
"attributes": {
"message": "OK"
}
}
}
  • /db_health: This endpoint is used to check the database connection status of service as well as providing some info about the service database, such as how many tables in the database and dealyed jobs info. Below is the expected JSON response:
{
"data": {
"type": "checker",
"id": "1",
"attributes": {
"connected": true,
"tables": "4",
"max_connection_size": "5",
"open_connection": "1",
"dj_total_count": "",
"dj_faild_count": "",
"dj_terminated_count": ""
}
}
}
  • /tcp_dependencies_health : This endpoint should report the status (either up or down) of all the TCP dependencies. The dependencies need to be configurable using environment variables that follow this naming convention TCP_DEPENDENCY_##. An example of these environment variables is shown below
# ${VAR_NAME}=${host_or_ip}:${port}
TCP_DEPENDENCY_00=mysql:6604
TCP_DEPENDENCY_01=redis:9089

Below is the expected JSON response for a service configured with the above variables :

{
"data": {
"type": "checker",
"id": "1",
"attributes": {
"mysql_6604": {
"message": "UP"
},
"redis_9089": {
"message": "DOWN"
}
}
}
}
  • /http_dependencies_health : This endpoint should report the status of all the HTTP dependencies. The response of this endpoint will include the status code of each of the configured dependencies as well as the body of the HTTP health check response. The HTTP dependencies need to be configurable for each of the services using environment variables that follow this naming convention HTTP_DEPENDENCY_##. An example of these environment variables is shown below
#${VAR_NAME}=http://${host_or_ip}:${port}/${health_endpoint}
HTTP_DEPENDENCY_00=http://localhost:3000/health
HTTP_DEPENDENCY_01=http://localhost:80/health
HTTP_DEPENDENCY_02=http://localhost:3090/health

Below is the expected JSON response for a service configured with the above variables :

{
"data": {
"type": "checker",
"id": "1",
"attributes": {
"http://localhost:3090/health": {
"status": null,
"details": "Failed to open TCP connection to localhost:3090"
},
"http://localhost:3000/health": {
"status": 200,
"details": {
"data": {
"type": "checker",
"id": "1",
"attributes": {
"message": "OK"
}
}
}
},
"http://localhost:80/health": {
"status": 200,
"details": ""
}
}
}
}

Implementation

I implemented the above features and published a rack middleware gem that encapsulate the described health check endpoints. You can find the gem on Github and Rubygems. Integrating and using the gem with rack and rails applications is very simple and can be done by performing the following steps.

  • Adding the gem to rails applications can be done by adding the below line to the Gemfile
gem 'deep_health_check'
  • Use the middleware in the application by adding the below line to the application.rb for rails and for rack to the application class.
config.middleware.insert_after Rails::Rack::Logger, DeepHealthCheck::MiddlewareHealthCheck
  • Configure the TCP and HTTP dependencies via environment variables. The middleware can handle up to 100 dependencies for each of the TCP and HTTP dependencies. The TCP dependencies should follow the following format ${host_or_ip}:${port} while the HTTP dependencies should be a valid HTTP URL. Below is an example of these configurations.
TCP_DEPENDENCY_00=127.0.0.0:8080
TCP_DEPENDENCY_01=127.0.0.0:8080
HTTP_DEPENDENCY_00=http://127.0.0.0:8080/health
HTTP_DEPENDENCY_01=http://127.0.0.0:8080/health

The above environment variables need to be available to the running instances of the application and usually, these configs can be stored on the servers running the applications with all other configurations such as the database URLs. And in the case of containers, these environment variables can be passed to the containers during the deployment time.

Protect the health check endpoints

Some of the middleware supported endpoints provide information about the database or the application's dependencies. This information should not be exposed to the public because it may introduce security risks for the application and make it vulnerable to attacks. Therefore, there is a need to protect these endpoints to reduce the security risk introduced by exposing them. This task can be achieved simply by protecting these endpoints using basic auth credentials (of course there are other options πŸ˜ƒ). The following page provides all the necessary steps needed to achieve this task using basic auth credentials.

Conclusion

Health check endpoints are useful for may purposes and use cases such as monitoring or zero downtime software deployments. The health checks need to be implemented for most of the applications, this post described one way for implementing and integrating the health checks endpoints for ruby/rack applications.

The Middleware gem can be found both on Github and Rubygems.

Follow us on Twitter 🐦 and Facebook πŸ‘₯ and Instagram πŸ“· and join our Facebook and Linkedin Groups πŸ’¬.

To join our community Slack team chat πŸ—£οΈ read our weekly Faun topics πŸ—žοΈ, and connect with the community πŸ“£ click here⬇

If this post was helpful, please click the clap πŸ‘ button below a few times to show your support for the author! ⬇

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts. Medium’s largest DevOps publication.

Al-Waleed Shihadeh

Written by

Team Lead & Product Owner

Faun

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts. Medium’s largest DevOps publication.

More From Medium

More from Faun

More from Faun

BERT for Everyone

Andre Ye
Feb 28 Β· 5 min read

158

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium β€” and support writers while you’re at it. Just $5/month. Upgrade