OpenPaaS Story: Health Check API

Photo by Hüseyin on Unsplash

After deploying OpenPaaS for production, the health status of our application is one big thing we need to keep our eyes on. Is OpenPaaS online and running? Does every service linked to OpenPaas work correctly? We want to ensure that not only our core OpenPaaS but also our services are operating correctly. Hence, developing a Health Check API is a must. This API will report all current statuses of services and make life simpler for our DevOps team.

Challenge

A lot of services linked to OpenPaaS, and they can all be enabled/disabled easily. Also, each of them has a different way of checking the state of health or connection. Building the Health Check API is therefore quite challenging, as we can not have a hard-coded set of services that we want to perform a health check.

Solution

From the OpenPaaS characteristic, we came up with an appropriate solution. We will create a health check platform for OpenPaaS, where each service can register for its own health check-up once the service is enabled and initiated. This way we can monitor dynamically all the resources we want to conduct a health test. If the service is disabled, it will not be listed on our health check platform and, of course, there will be no health check-up.

A health check provider will contain the name of the service and a function called checker. checker will return a Promise, which will resolve a formatted message containing information about the status of the service. This function will be called every time we want to retrieve the health status of the provider, returning the health data at that exact moment.

The creation of a centralized health check platform also benefits us in controlling the way each service checks its status. Since each service has a different method of checking the state of the connection, individual registration will make it easier for us to set or change that method without touching the core part or changing too much code in the future.

Figure 0. OpenPaaS and some of our services as health check providers

Implement the core health check platform

There seems to be a lot of works to be done when it comes to implementing a health check feature. According to the solution above, we need to create a health registry factory, where each service can register for its own health check-up. Here is a list of stuff we need to do:

1. Create a health check provider model.
2. Implement a way to register and retrieve service as a provider.
3. Define the returned message in a formatted way.
4. Register services as providers once they get initiated.

Let’s do it!

First of all, we have to define our health check provider. Each service connects to OpenPaaS is a provider. To keep this simple, our provider will have two mandatory properties, name & checker, which represent the name of the provider and the method function used to obtain the health status of the service.

Figure 1. HealthCheckProvider

For every service registered on our core health check platform, we will create a new instance of HealthCheckProvider. Now what we need to do is create a place to store all of these providers, and some methods to register or retrieve them. We defined an empty object called providers, which we used to store our registered services and two methods, register for signing up new providers, and getCheckers for retrieving our providers and their checkers.

Figure 2. Register & Retrieve functions

The next thing we need to do is create a generic format for our messages. A message will contain at least status, name of the service, and details which contain advanced information about it. This one should not be a big problem.

Figure 3. HealthCheckMessage

The status of the service can be healthy or unhealthy, depends on the connection status from OpenPaaS to it. For now, our health check platform is pretty completed, we will register some services to our platform when they are initialized. Here is an example with MongoDB:

Figure 4. MongoDB Health Check

Note that checkConnection is a function we used to retrieve the status of MongoDB connection. Different services will have different methods to check for their own statuses. The register function will be called once OpenPaaS initiates the connection to MongoDB.

API Endpoints

Now that the core platform for health check has been completed, the next job is to expose some APIs so that our administrators, the DevOps team, or some external probes can monitor OpenPaaS status at any time. We are going to create three endpoints:

• /healthcheck: Retrieve the statuses of all services.
• /healthcheck/{name}: Get a single service by name.
• /healthcheck/services: Get all available services for performing health checks.

These APIs may contain some sensitive data. It is therefore necessary to perform some authorization checks before allowing users to retrieve the information.

Writing the APIs is simple. But the question is: “How can we access and get the status of our registered services?”.
The idea is very clear. For the hard part, we query for the corresponding provider according to the name, get the checker function ( which is a Promise ) then run it. In case we want to retrieve the statuses of all services, we get all the available services inside our health check platform. The result will be in a formatted message as declared above, contain the state of the service’s connection at the moment the API is called. Here are some codes to explain in detail how we do it:

Figure 5. Getting data

The next step is to return the message as the response for our API.

Here is the result:

Figure 6. API Response

You can find more about our health check APIs via our documentation.

Things we need to improve and update

We are now having a good health check platform, but still, there are so many things left to improve and update in the future. Here are some of the things we have thought about:

• Calculate and report the latency of any service connection
• The API response includes clearer error messages when things go wrong
• Track and report if a huge number of transactions are taking place.

Whenever we think we are good, we can be even better.

Summary

In this article, we showed you how we implement Health Checks Platform for OpenPaaS.
Although there are so many more things to do with monitoring, I hope this article gives you an idea of the challenges we have encountered, how we have managed to find and implement the appropriate solution. A lot of things have been waiting for us to be implemented and improved, and we can’t wait to solve these interesting challenges.
Would you like to be a part of us and make awesome things? Here’s the key to your treasure.