Microservices Monitoring with Health Checks using WatchDog

In this article, we are going to Developing Microservices Monitoring with Health Checks using WatchDog. We will Use the HealthChecks feature in our back-end ASP.NET microservices.

https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/monitor-app-health

Microservice architecture have become the new model for building modern cloud-native applications. And microservices-based applications are distributed systems.

While architecting distributed microservices-based applications, it get lots of benefits like makes easy to scale and manage services, but as the same time, it is increasing interactions between those services have created a new set of problems.

So we should assume that failures will happen, and we should dealing with unexpected failures, especially in a distributed system. For example in case of network or container failures, microservices must have a strategy to retry requests again.

How do you handle the complexity that comes with cloud and microservices?

— Health Monitoring
Microservice should design for monitoring with health monitoring, Health monitoring is critical to multiple aspects of operating microservices.
By this way, we can understand for a particular microservice is alive and ready to accommodate requests. We can also provide health information to our orchestrator’s cluster, so that the cluster can act accordingly. For example Kubernetes has Liveness and Readiness probes that we can address health check urls.

That make a good health reporting which customized for our microservices
like adding sub health checks for underlying database connection,
and by this way we can detect and fix issues for your running application much more easily.

You can see in first picture, for Ordering microservices health checks,
it has 3 sub health checks

• 1 for self ordering microservices
• 2 for underlying database connection,
• 3 for underlying rabbitmq connection

So since all sub conditions healthy, after that we can say ordering microservices is ready to accommodate requests.

Let’s check our big picture and see what we are going to build one by one.
As you can see that, we are in here and start to developing “Microservices Monitoring with Health Checks using WatchDog”.

We are going to cover the;

• Implement health checks in ASP.NET Core services
• Use the HealthChecks feature in your back-end ASP.NET microservices
• This will be the aspnet health check implementation with custom health check methods which includes database availabilities — for example in basket microservices, we will add sub-hc conditions for connecting Redis and RabbitMQ.
• Query microservices to report about their health status
• Returning health status in JSON format
• Use Watchdogs which is separate service that can watch health and load across microservices, and report health about the microservices by querying with the HealthChecks methods.

We will apply these microservices Monitoring Health Checks for all microservices starting with Catalog microservice and continue to add for all microservices. After that we will create a new Microservices which will be “WebStatus”. This will implement health check UI and listen all microservices health checks and visualize with WatchDog. Lastly we containerize for all microservices on docker environment and monitoring healths with checking “WebStatus” microservice.

Background

This is the introduction of the series. This will be the series of articles. You can follow the series with below links.

• 0- Microservices Observability, Resilience, Monitoring on .Net
• 1- Microservices Observability with Distributed Logging using ElasticSearch and Kibana
• 2- Microservices Resilience and Fault Tolerance with applying Retry and Circuit-Breaker patterns using Polly
• 3- Microservices Monitoring with Health Checks using WatchDog

We will focus on microservices cross-cutting concerns on these article series.

Step by Step Development w/ Udemy Course

Get Udemy Course with discounted — Microservices Observability, Resilience, Monitoring on .Net.

Source Code

Get the Source Code from AspnetRun Microservices Github — Clone or fork this repository, if you like don’t forget the star. If you find or ask anything you can directly open issue on repository.

Asp.Net Health Checks

When you are working with Microservices, Health monitoring is critical to multiple aspects of operating microservices. Unhealthy services leads to a lot of revenue loss, customer confidence, and negative effects on the business. In order to avoid such situations, we should regularly monitor the status of our apps.

Using these health checks, we can be notified to customers about outages beforehand and reduce the impact of the outage by repairing it or notifying the customer about the ongoing outage, leading to trust.
While a simple application may require monitoring of its dependencies once a day, a critical application may need to be monitored as often as possible.
We are going to provide a status page to view results and add functionality to inform developers about their issues.

We will use the health check API endpoint, which will return the instant health state (Healthy / Unhealthy / Degraded) of each microservice in response to us. This endpoint will perform the necessary health check checks that we define.

WatchDog Health Dashboard

First, let’s try to explain the health status we mentioned above:
Healthy: All of our ingredients are healthy.
Unhealthy: There may be a problem with at least one of our components or an unhandled exception may have been thrown during the health check.
Degraded: This indicates that the components are all healthy but the controls are slow or unstable. We get a successful response with a simple database query, but the execution time is higher than we expected.

Ok now we can talk about implementing basic health monitoring when developing ASP.NET Core Microservices, you can use a built-in health monitoring feature by using a nuget package “Microsoft.Extension.Diagnostic.HealthCheck”.
These health monitoring features can be enabled by using a set of services and middleware.

Go to ->
https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks#ui-storage-providers

See Health Checks
HealthChecks packages include health checks for:

Sql Server
MySql
Oracle
Sqlite
RavenDB
Postgres
EventStore
…

See HealthCheckUI
The project HealthChecks.UI is a minimal UI interface that stores and shows the health checks results from the configured HealthChecks uris.
To integrate HealthChecks.UI in your project you just need to add the HealthChecks.UI services and middlewares available in the package: AspNetCore.HealthChecks.UI

So now we are ready to implement health checks.

Adding Health Check for Catalog.API Microservices with Checking MongoDb Connection

Before we start, we should talk about the HealthChecks nuget packages,

We are not installment required package, it comes from aspnet framework

Nuget Package
Install-Package Microsoft.AspNetCore.Diagnostics.HealthChecks

After that, We are going to;
— Add Health Check

Startup.cs -> 
 
ConfigureServices
services.AddHealthChecks();
Configure
 app.UseEndpoints(endpoints =>
 {
 endpoints.MapControllers();
 endpoints.MapHealthChecks(“/hc”); — → ADDED
 });

Now we can

• TEST:
  http://localhost:5000/hc
  Healthy

But for Catalog.API project using mongodb and need to check mongodb connection is ready or not. In case of Catalog.API is working but mongodb container is not running, That means Catalog.API web application can be worked but some of some components not working. In that cases we set Degraded as a Health Check result.

In order to check mongo connection in HealthCheck

Nuget Package :
Install-Package AspNetCore.HealthChecks.MongoDb

Add new sub item

Startup.cs -> ConfigureServices
services.AddHealthChecks()
 .AddMongoDb(Configuration[“DatabaseSettings:ConnectionString”], “Catalog MongoDb Health”, HealthStatus.Degraded);

Now we can test Catalog hc with mongo db.
Stop mongo docker images and test again

docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d
docker-compose -f docker-compose.yml -f docker-compose.override.yml down

• TEST:
  http://localhost:5000/hc
  Degraded

Start mongo docker images and test again

docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d
docker-compose -f docker-compose.yml -f docker-compose.override.yml down

• TEST: — after run mongo
  http://localhost:5000/hc
  Healthy

Adding json response

Nuget Package
Install-Package AspNetCore.HealthChecks.UI.Client

Startup.cs -> Configure
app.UseEndpoints(endpoints =>
 {
 endpoints.MapControllers(); 
 endpoints.MapHealthChecks(“/hc”, new HealthCheckOptions()
 {
 Predicate = _ => true,
 ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
 });
 });

TEST:

• Run the application
  Change Run Profile to “Catalog.API” and Run the application
• http://localhost:5000/hc

JSON Response :

{“status”:”Healthy”,”totalDuration”:”00:00:00.0023926",”entries”:{“MongoDb Health”:{“data”:{},”duration”:”00:00:00.0021199",”status”:”Healthy”,”tags”:[]}}}

As you can see that, we have added Health Check for Catalog.API Microservices with Checking MongoDb Connection.

For Basket Microservices, we should add Redis health check.

Add Redis HC
In order to check redis connection in HealthCheck

Nuget Packages :
Install-Package AspNetCore.HealthChecks.Redis

Startup.cs -> ConfigureServices
services.AddHealthChecks() — — — — — — — → ADDED
 .AddRedis(Configuration[“CacheSettings:ConnectionString”], “Redis Health”, HealthStatus.Degraded);

For Shopping.Aggregator and AspnetRunBasics microservices, we can add UrlGroup in order check sub healths.

Add Uri Group HC for Shopping.Aggregator and AspnetRunBasics

Nuget Package :
Install-Package AspNetCore.HealthChecks.Uris

Shopping.Aggregator

Startup.cs -> ConfigureServices
services.AddHealthChecks()
 .AddUrlGroup(new Uri($”{Configuration[“ApiSettings:CatalogUrl”]}/swagger/index.html”), “Catalog.API”, HealthStatus.Degraded)
 .AddUrlGroup(new Uri($”{Configuration[“ApiSettings:BasketUrl”]}/swagger/index.html”), “Basket.API”, HealthStatus.Degraded)
 .AddUrlGroup(new Uri($”{Configuration[“ApiSettings:OrderingUrl”]}/swagger/index.html”), “Ordering.API”, HealthStatus.Degraded);

Developing WebStatus App for Centralized Microservices Health Monitoring Using Watchdogs for Visualization

First of all, We are going to start with;

• Create New Project
  AspNet MVC Project
  Name
  WebStatus
• Set Port
  http://localhost:5007
• Set a Startup Porject
  Select “WebStatus” Project
  Run Application

After that, install required nuget packages

Add Nuget Package

Install-Package AspNetCore.HealthChecks.UI
Install-Package AspNetCore.HealthChecks.UI.InMemory.Storage

Manage Startup for Adding HealthChecksUI with Watchdog

Startup.cs

public void ConfigureServices(IServiceCollection services)
{
 //…
 // Registers required services for health checks
 services.AddHealthChecksUI()
 .AddInMemoryStorage();
}
//…
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
 //…
 app.UseEndpoints(endpoints =>
 {
 endpoints.MapHealthChecksUI(); — — — -> ADDED
 //…
}

As you can see that, we have Developing WebStatus App for Centralized Microservices Health Monitoring.
After adding packages and startup developments, We should configure our Health Monitoring uris in order to check hc results

Configuration file for health check UI:

appsettings.json

“HealthChecks-UI”: {
 “HealthChecks”: [
 {
 “Name”: “Catalog Health Check”,
 “Uri”: “http://localhost:5000/hc”
 },
 {
 “Name”: “Basket Health Check”,
 “Uri”: “http://localhost:5001/hc”
 }
 ]
 },

For testing purpose, this time I am adding only Catalog and Basket hc.
But after test I will add for all microservices with docker urls.

Of course we need to
— Redirect Home Page to HC

HomeController
public IActionResult Index()
 {
 return Redirect(“/healthchecks-ui”); — — — → ADDED
 //return View();
 }

Default UI Path comes below library — https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks/blob/master/src/HealthChecks.UI/Configuration/Options.cs
public string UIPath { get; set; } = “/healthchecks-ui”;

Now we can test the application

• Set Multiple Run Profile
  Catalog.API
  Basket.API
  WebStatus
• Run Application :

localhost:5007/healthchecks-ui

See dashboard worked !

Adding All Microservices HC urls with docker uris
— Add All HC uri s into Configuration

appsettings.json file

“HealthChecks-UI”: {
 “HealthChecks”: [
 {
 “Name”: “Catalog Health Check”,
 “Uri”: “http://localhost:8000/hc"
 },
 {
 “Name”: “Basket Health Check”,
 “Uri”: “http://localhost:8001/hc"
 },
 {
 “Name”: “Discount Health Check”,
 “Uri”: “http://localhost:8002/hc"
 },
 {
 “Name”: “Ordering Health Check”,
 “Uri”: “http://localhost:8004/hc"
 },
 {
 “Name”: “Shopping Aggregator Health Check”,
 “Uri”: “http://localhost:8005/hc"
 },
 {
 “Name”: “AspnetRunBasics WebMVC Health Check”,
 “Uri”: “http://localhost:8006/hc"
 }
 ],

We have configured for docker urls. But now that microservices docker images not included hc developments. So We will test with building again full images. Because we have added hc developments.

As you can see that, we have finished to Development of WebStatus App for Centralized Microservices Health Monitoring Using Watchdogs for Visualization.

Containerize WebStatus Health Monitoring Microservices using Docker Compose

We are going to Containerize WebStatus Health Monitoring Microservices using Docker Compose. We are going to add Docker-Compose File for WebStatus Application Microservices Solution.

First, We should go WebStatus project. In order to create DockerFile and docker compose file we use Visual Studio Container Orchestrator Support.

In the Discount.API project,
Right Click
choose Add > ..Container Orchestrator Support.
The Docker Support Options dialog appears.
Choose Docker Compose.
Choose Linux.

DockerFile and docker-compose created.

Visual Studio creates DockerFile and update the docker-compose.yml file with newly added WebStatus images.

Examine Docker-Compose yaml file

We should Format docker-compose yaml files with tabs. These yaml files is very strict to indentation of the commands.

docker-compose.yml

version: ‘3.4’
..
webstatus:
 image: ${DOCKER_REGISTRY-}webstatus
 build:
 context: .
 dockerfile: WebApps/WebStatus/Dockerfile

docker-compose.override.yml

version: ‘3.4’
…
webstatus:
 container_name: webstatus
 environment:
 — ASPNETCORE_ENVIRONMENT=Development
 — HealthChecksUI__HealthChecks__0__Name=Catalog Health Check
 — HealthChecksUI__HealthChecks__0__Uri=http://catalog.api/hc
 — HealthChecksUI__HealthChecks__1__Name=Basket Health Check
 — HealthChecksUI__HealthChecks__1__Uri=http://basket.api/hc
 — HealthChecksUI__HealthChecks__2__Name=Discount Health Check
 — HealthChecksUI__HealthChecks__2__Uri=http://discount.api/hc
 — HealthChecksUI__HealthChecks__3__Name=Ordering Health Check
 — HealthChecksUI__HealthChecks__3__Uri=http://ordering.api/hc
 — HealthChecksUI__HealthChecks__4__Name=Shopping Aggregator Health Check
 — HealthChecksUI__HealthChecks__4__Uri=http://shopping.aggregator/hc
 — HealthChecksUI__HealthChecks__5__Name=AspnetRunBasics WebMVC Health Check
 — HealthChecksUI__HealthChecks__5__Uri=http://aspnetrunbasics/hc 
 ports:
 — “8007:80”

Overriding existing appsettings on docker container. Overriding array appsettings on Docker we used double __ characters.

You can see that we have override ocelot HealthChecks api url with docker container names.

As a summary, we have created Dockerfile and docker-compose files for WebStatus Health Monitoring Application configuration.
So now we can test all microservices with in docker-compose environment.

Test on Docker environment — WebStatus Health Monitoring Microservices into Docker Compose for Visulize WatchDog HC

Close all dockers and run with below command on that location;

Start/Stop Docker compose :

docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d
 docker-compose -f docker-compose.yml -f docker-compose.override.yml down

• Check the WebStatus
• WebStatus — HC List
  http://localhost:8007
  http://host.docker.internal:8007

As you can see that, we have tested all microservices on docker environment with heath check features on WebStatus Health Monitoring Microservices.

Step by Step Development w/ Udemy Course

Get Udemy Course with discounted — Microservices Observability, Resilience, Monitoring on .Net.