Active health check using Openresty for HTTPS upstream in Nginx

When utilizing the community edition of Nginx open source as a front-facing load balancer, one of the primary challenges encountered is the absence of active health checks upstream. While this edition offers passive health checks, it becomes evident, that they are insufficient for production scenarios ( due to how it works ) where minimizing downtime is critical.

In this post, we’ll delve into the limitations of passive health checks and explore how leveraging Openresty can enable the integration of active health checks into applications without unnecessary cost for Ngnix Plus which offers active health checks OOB.

Limitations of passive health checks in Nginx

HTTP server is marked as unhealthy during passive health check due to the below reasons.

1. If there is an error while establishing a connection to the server.
2. The server is not responding at all
3. Timeout while establishing a connection
4. The response header is empty.

So, is API returning 500 response code considered as an unhealthy server? ( No, because the server is healthy but the application is not.) Round robin algorith will keep sending requests to this unhealthy server and keep failing.

There are some additional fields to handle the server-down scenario like setting up failure count, timeout, etc. But the point here is after timeout only the way to check if the server is healthy or not is by sending the requests to the server, which helps in some bit by not sending requests to the unhealthy server for a specified timeout but doesn't give full control to the situation and keeps failing requests by sending traffic to the unhealthy server.

While in production use cases we need more than this, we want to have more granular control on the definition of unhealthy, send traffic to only healthy servers, do asynchronous checks to check server health and remove unhealthy servers until they become healthy again.

Active health checks using Openresty

While there are open source libraries/plugins available for Nginx to have active health checks, I found openresty is a good solution as it has an inbuilt Lua engine and built-in blocks that do all of this job very efficiently without impacting performance.

So, How can you proceed with using openresty along with Nginx existing configurations? In load balancer.

Step 1

Install the openresty, It has covered good documentation on installation steps for various distros.

#Test the Installation 
resty -v

Step 2

Configure, the main Nginx configuration file which is Nginx.conf will be available at openresty path (/usr/local/openresty/ngnix/conf/nginx.conf ), this file will have all required configuration from Ngnix original configuration (/etc/nginx/nginx.conf) and it does also include the site paths to include default configs.

Step 3

Health check block, Add the health check block as below in your sites-enabled/default file or Nginx.conf file depending on your configuration.

lua_shared_dict healthcheck 1m;

upstream backend {
     server foo.com:443;
     server bar.com:443;
}

init_worker_by_lua_block {
        local hc = require "resty.upstream.healthcheck"

        local ok, err = hc.spawn_checker{
            shm = "healthcheck",
            upstream = "backend",
            type = "https",

            http_req = "GET /v1/api/health HTTP/1.0\r\nHost: backend\r\n\r\n",

            port = nil,
            interval = 5000,
            timeout = 1000,
            fall = 3,
            rise = 2,
            valid_statuses = {200, 302},
            concurrency = 10,
        }
        if not ok then
            ngx.log(ngx.ERR, "failed to spawn health checker: ", err)
            return
        end
    }

In the above example, we have two upstream servers foo.com and bar.com which are serving on HTTPS and the port is 443. We want to have active health checks for both of these servers which are load balancing the traffic.

The init_worker_by_lua_* block triggers the health check asynchronously on the configuration provided in the block, we have to specify the interval in ms at which this check will be executed.

The http_req provided in the block will be called a GET request for the health endpoint provided. This endpoint has to be accessible on the upstream server.

It gives the ability to implement the application-specific health functionality which could be more complex if your application API is using multiple microservices for a functionality. Valid statuses can also be customized on response codes that come from health API.

Step 4

Start the openresty, you need to stop and disable the nginx service to start serving traffic from openresty and avoid conflicts.

service start openresty.service

It completes the switching traffic from Nginx to Openresty.