Elasticsearch: health check on .Net application

How often have you encountered exceptions while starting up your application using docker-compose?

  Error Message:
   Elastic.Transport.TransportException : No connection could be made because the target machine actively refused it. (localhost:9200). Call: Status code unknown from: HEAD /index
---- System.Net.Http.HttpRequestException : No connection could be made because the target machine actively refused it. (localhost:9200)
-------- System.Net.Sockets.SocketException : No connection could be made because the target machine actively refused it.
  Stack Trace:
     at Elastic.Transport.DefaultHttpTransport`1.HandleTransportException(RequestData data, Exception clientException, TransportResponse response)
   at Elastic.Transport.DefaultHttpTransport`1.FinalizeResponse[TResponse](RequestData requestData, RequestPipeline pipeline, List`1 seenExceptions, TResponse response)
   at Elastic.Transport.DefaultHttpTransport`1.Request[TResponse](HttpMethod method, String path, PostData data, RequestParameters requestParameters)
   at Elastic.Clients.Elasticsearch.ElasticsearchClient.DoRequest[TRequest,TResponse,TRequestParameters](TRequest request, Action`1 forceConfiguration) in /_/src/Elastic.Clients.Elasticsearch/Client/ElasticsearchClient.cs:line 147
   at Elastic.Clients.Elasticsearch.NamespacedClientProxy.DoRequest[TRequest,TResponse,TRequestParameters](TRequest request, Action`1 forceConfiguration) in /_/src/Elastic.Clients.Elasticsearch/Client/NamespacedClientProxy.cs:line 44
   at Elastic.Clients.Elasticsearch.NamespacedClientProxy.DoRequest[TRequest,TResponse,TRequestParameters](TRequest request) in /_/src/Elastic.Clients.Elasticsearch/Client/NamespacedClientProxy.cs:line 32
   at Elastic.Clients.Elasticsearch.IndexManagement.IndicesNamespacedClient.Exists(Indices indices) in /_/src/Elastic.Clients.Elasticsearch/_Generated/Client/ElasticsearchClient.Indices.g.cs:line 1463

By checking the error you quickly discover that your database was not ready when you first connected.

version: "3.3"

services:
  app:    
    container_name: app
    build:      
      context: ./App
      dockerfile: ./Dockerfile 
    networks:
      - app_net
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.9.1
    restart: unless-stopped
    environment:
      - xpack.security.enabled=false
      - xpack.security.http.ssl.enabled=false
      - xpack.security.transport.ssl.enabled=false
    ports:
      - 9200:9200
    networks:
      - app_net

networks:
  app_net:

We try repeatedly but notice that our database will never be ready before the first connection.

A reindeer surprised by the exception

Fortunately, we have many strategies that allow us to check the status of our databases before querying them.

Docker Healthcheck

In the case of docker-compose, there are some exciting techniques to implement. In fact, there is the option to utilize a specific feature known as HealthCheck:

https://docs.docker.com/engine/reference/builder/#healthcheck

Thanks to this mechanism we can indicate which services depend on others and also specify what the command is to verify that the service is active and ready to receive connections.

version: "3.3"

services:
  app:    
    container_name: app
    build:      
      context: ./App
      dockerfile: ./Dockerfile 
    depends_on:
      elasticsearch:
        condition: service_healthy
    networks:
      - app_net
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.9.1
    restart: unless-stopped
    environment:
      - xpack.security.enabled=false
      - xpack.security.http.ssl.enabled=false
      - xpack.security.transport.ssl.enabled=false
    healthcheck:
     test:
       [
         "CMD-SHELL",
         "curl -s http://elasticsearch:9200 >/dev/null || exit 1",
       ]
     interval: 10s
     timeout: 10s
     retries: 120
    ports:
      - 9200:9200
    networks:
      - app_net
   
networks:
  app_net:

Thanks to the healtcheck we can indicate which test to perform to check when Elasticsearch is active and ready.

Here is a more complex example with Elasticsearch secured by a self-signed SSL certificate:

elastic-stack-docker-part-one/docker-compose.yml at main · elkninja/elastic-stack-docker-part-one

healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]

In-app Healthcheck

There are situations where we do not have the possibility of using the docker healcheck mechanism and in fact, if we have a bit of experience with other services (for example Kibana) we know that it is important to have a programmatic check at startup to verify the ElasticSearch service status.

But how can we perform this check in our .Net applications?

Very simple, Elasticsearch provides us with various APIs to understand the status of the node: https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html

First, we need to introduce options that allow us to control whether and how to perform this check:

public class ElasticsearchDbOptions
{
    public string ConnectionString { get; set; } = null!;
    public bool Debug { get; set; }
    public ElasticsearchPingDbOptions Ping { get; set; } = new ElasticsearchPingDbOptions();
}

public class ElasticsearchPingDbOptions
{
    public bool Enabled { get; set; } = true;
    public int Retries { get; set; } = 10;
    public TimeSpan Interval { get; set; } = TimeSpan.FromSeconds(30);
}

We can then use the Ping method present both on the NEST 7.17.x package and on the new .NET client for Elasticsearch 8.x .

public class ElasticsearchDbContext : IElasticsearchDbContext
{
    protected readonly ILogger<BaseElasticsearchDbContext> _logger;
    protected readonly BaseElasticsearchDbOptions _dbOptions;
    protected static bool _pingExecuted;
    public ElasticsearchClient Client { get; }

    public BaseElasticsearchDbContext(IOptions<BaseElasticsearchDbOptions> dbOptions, ILogger<BaseElasticsearchDbContext> logger)
    {
        _dbOptions = dbOptions.Value;
        _logger = logger;

        var clientSettings = new ElasticsearchClientSettings(new Uri(_dbOptions.ConnectionString))
         .ThrowExceptions();

        if (_dbOptions.BasicAuthentication)
        {
            if (_dbOptions.UserName != null && _dbOptions.Password != null)
            {
                clientSettings.Authentication(new BasicAuthentication(_dbOptions.UserName, _dbOptions.Password));
            }
            else
            {
                throw new Exception("BasicAuthentication enabled with no UserName and Password.");
            }
        }

        if (_dbOptions.Debug)
        {
            clientSettings.EnableDebugMode().DisableDirectStreaming().PrettyJson();

            clientSettings.OnRequestCompleted(apiCallDetails =>
            {
                var requestBody = apiCallDetails.RequestBodyInBytes != null ? Encoding.UTF8.GetString(apiCallDetails.RequestBodyInBytes) : "No body";
                _logger.LogDebug("Request method: {@httpMethod} Request path: {@uri} Request: {@request}", apiCallDetails.HttpMethod, apiCallDetails.Uri.PathAndQuery, requestBody);
            });
        }

        Client = new ElasticsearchClient(clientSettings);

        if (!_pingExecuted && _dbOptions.Ping.Enabled)
        {
            Ping();
            _pingExecuted = true;
        }
    }
  
.............
.............

}

And finally the Ping method:

protected virtual void Ping()
{
    int attempts = 1;
    while (true)
    {
        try
        {
            _logger.LogInformation("Pinging ElasticSearch");
            var pingResult = Client.Ping();
            if (pingResult.IsValidResponse)
            {
                _logger.LogInformation("Elasticsearch pinged successfully");
                break;
            }
            else
            {
                if (pingResult.TryGetOriginalException(out var ex))
                {
                    _logger.LogWarning(ex, "Unable to connect to Elasticsearch");
                }
                else
                {
                    _logger.LogWarning("Unable to connect to Elasticsearch");
                }
            }
        }
        catch (Exception ex)
        {
            _logger.LogWarning(ex, "Unable to connect to Elasticsearch");
        }
        if (attempts > _dbOptions.Ping.Retries)
        {
            _logger.LogError("Maximum number of attempts exceeded, unable to connect to Elasticsearch");
            throw new Exception("Maximum number of attempts exceeded, unable to connect to Elasticsearch");
        }
        else
        {
            _logger.LogWarning("Attempt {@attempts} of {@retries}, retry in {@interval}", attempts, _dbOptions.Ping.Retries, _dbOptions.Ping.Interval);
            Thread.Sleep(_dbOptions.Ping.Interval);
        }
        attempts++;
    }
}

As you can see from the code I use static named _pingExecutedto not repeat the ping every time the DbContext is instantiated but only when the application is in the startup phase.

PS: If the Elasticsearch security features are enabled, you must have a few permissions to use API. https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#cluster-health-api-prereqs

PS: I use the same approach for Microsoft SQL Server and Oracle Database, if there is interest I could publish specific articles. Tell me in the comments section.

Please, Subscribe and Clap! Grazie!