Distributed Logging with Elasticsearch and Kibana
Did you know that up to 30% of software development time is spent debugging code? In this article, we’ll show you how to save time and effort by automating the error-catching process and logging errors to Elasticsearch.
Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. And through it, you can store logs of other types, according to your needs in the project. This powerful tool allows you to easily view and analyze system-level errors later on Kibana, a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack.
Kibana lets you do anything from tracking query load to understanding the way requests flow through your apps. To enable logging with Elasticsearch in .NET, we’ll be using Serilog, a plugin which provides diagnostics logging to files, console, and elsewhere. Various sinks are available for Serilog which we can set up easily has a clean API.
Before we begin, let’s make sure to add the necessary Nget packages to our project. This step is crucial as it ensures that our project has access to the required libraries and dependencies to function properly. Without these packages, our project may encounter errors and bugs, which can impact its overall performance and functionality. By adding the Nget packages beforehand, we can avoid these issues and ensure a smooth and seamless development process.
Library we needed:
1- Serilog
2- Serilog.Sinks.Elasticsearch
3. Serilog.AspNetCore
4. Serilog.Enrichers.Environment
All these libraries is working together for catch and storage exception in Elasticsearch.
Project Contents:
1 — ErrorResponseModel.cs
public class ErrorResponseModel
{
public string? TraceId { get; set; }
public string? Type { get; set; }
public string? Title { get; set; }
public int? Status { get; set; }
public string? Message { get; set; }
}The error response class represents the final result that will be displayed in the frontend
2 — ExceptionHandlingMiddleware.cs
public class ExceptionHandlingMiddleware
{
private readonly RequestDelegate Next;
private readonly ILogger<ExceptionHandlingMiddleware> Logger;
public ExceptionHandlingMiddleware(RequestDelegate next, ILogger<ExceptionHandlingMiddleware> logger)
{
Next = next;
Logger = logger;
}
public async Task InvokeAsync(HttpContext httpContext)
{
try
{
await Next(httpContext);
}
catch (Exception ex)
{
Logger.LogError(ex, "Critical Exception");
await HandleExtension.HandleExceptionAsync(httpContext, ex);
}
}Exception Handling Middleware will handle all the requests came to the Api if we have any error it will catch an error here and store it and then send to the HandleExceptionAsync() function to identify the type of exception and return the exception information in the response of the request.
3 — HandleExtension.cs
namespace Distributed.Logging.Extensions
{
public static class HandleExtension
{
public static async Task HandleExceptionAsync(HttpContext context, Exception exception)
{
context.Response.ContentType = "application/json";
var response = context.Response;
var errorResponse = new ErrorResponseModel
{
Status = context.Response.StatusCode,
TraceId = context.TraceIdentifier // IS REQUEST ID ALSO
};
switch (exception.GetType().ToString())
{
case "System.IndexOutOfRangeException":
response.StatusCode = (int)HttpStatusCode.NotFound;
errorResponse.Type = response.ContentType;
errorResponse.Title = nameof(HttpStatusCode.NotFound);
errorResponse.Status = response.StatusCode;
errorResponse.Message = exception.Message;
break;
case "Microsoft.EntityFrameworkCore.DbUpdateException":
response.StatusCode = (int)HttpStatusCode.NotAcceptable;
errorResponse.Type = response.ContentType;
errorResponse.Status = (int)HttpStatusCode.NotAcceptable;
errorResponse.Title = "Database Issues";
var dbUpEx = (Microsoft.EntityFrameworkCore.DbUpdateException)exception;
if (dbUpEx.InnerException is not null)
{
response.StatusCode = (int)HttpStatusCode.NotAcceptable;
errorResponse.Status = (int)HttpStatusCode.NotAcceptable;
errorResponse.Type = response.ContentType;
if (dbUpEx.InnerException.Message.ToLower().Contains("unique index"))
{
errorResponse.Title = "Unique Constraint";
errorResponse.Message = "Cannot insert duplicate key row";
}
else if (dbUpEx.InnerException.Message.ToLower().Contains("check Constraint"))
{
errorResponse.Title = "Check Constraint";
errorResponse.Message = "Constraint chech violation.";
}
else if (dbUpEx.InnerException.Message.ToLower().Contains("foreign key constraint"))
{
errorResponse.Title = "FOREIGN KEY constraint";
errorResponse.Message = "The INSERT statement conflicted with the FOREIGN KEY constraint";
}
else
{
errorResponse.Message = dbUpEx.Message;
}
}
break;
case "System.UnauthorizedAccessException":
response.StatusCode = (int)HttpStatusCode.Unauthorized;
errorResponse.Type = response.ContentType;
errorResponse.Title = nameof(HttpStatusCode.Unauthorized);
errorResponse.Status = response.StatusCode;
errorResponse.Message = exception.Message;
break;
case "System.ApplicationException":
exception.GetType();
if (exception.Message.Contains("Invalid token"))
{
response.StatusCode = (int)HttpStatusCode.Forbidden;
errorResponse.Type = response.ContentType;
errorResponse.Title = nameof(HttpStatusCode.Forbidden);
errorResponse.Status = response.StatusCode;
errorResponse.Message = exception.Message;
break;
}
response.StatusCode = (int)HttpStatusCode.BadRequest;
errorResponse.Type = response.ContentType;
errorResponse.Status = response.StatusCode;
errorResponse.Message = exception.Message;
break;
case "System.Collections.Generic.KeyNotFoundException":
errorResponse.Type = response.ContentType;
errorResponse.Title = nameof(HttpStatusCode.NotFound);
errorResponse.Status = response.StatusCode;
errorResponse.Message = exception.Message;
break;
case "System.IO.FileNotFoundException":
errorResponse.Type = response.ContentType;
errorResponse.Title = nameof(HttpStatusCode.NotFound);
errorResponse.Status = response.StatusCode;
errorResponse.Message = exception.Message;
break;
case "System.DllNotFoundException":
errorResponse.Type = response.ContentType;
errorResponse.Title = nameof(HttpStatusCode.NotFound);
errorResponse.Status = response.StatusCode;
errorResponse.Message = exception.Message;
break;
default:
response.StatusCode = (int)HttpStatusCode.InternalServerError;
errorResponse.Type = response.ContentType;
errorResponse.Title = nameof(HttpStatusCode.InternalServerError);
errorResponse.Status = response.StatusCode;
errorResponse.Message = "Internal Server errors. Check Logs!";
break;
}
await context.Response.WriteAsync(JsonSerializer.Serialize(errorResponse));
}
}
}This is class that provides an extension method for handling exceptions that might occur in an ASP.NET Core web application. It contains a static method called “HandleExceptionAsync” that takes in two parameters: the HttpContext object and an Exception object.
The method sets the Content Type of the response to “application/json” and creates an instance of the ErrorResponseModel class. It then checks the type of the exception and handles it accordingly.
For example, if the exception is an IndexOutOfRangeException, it sets the HTTP status code to 404 (NotFound) and populates the ErrorResponseModel with information about the error. Similarly, if the exception is a DbUpdateException, it sets the HTTP status code to 406 (NotAcceptable) and populates the ErrorResponseModel with information about the error.
The method then serializes the ErrorResponseModel object and writes it to the response. This allows the client to receive a JSON object with information about the error that occurred.
4 — SeriLoggerUtility.cs
public static class SeriLoggerUtility
{
public static Action<HostBuilderContext, LoggerConfiguration> Configure =>
(hostContext, configuration) =>
{
string Environment = configuration["ElasticConfiguration:Environment"];
var IndexFormat = $"app-logs-{Assembly.GetEntryAssembly().GetName().Name.ToLower().Replace(".", "-")}-{hostEnvironment.EnvironmentName.ToLower().Replace(".", "-")}-logs-{DateTime.UtcNow:yyyy-MM}";
if (Environment == "Online")
IndexFormat = $"app-logs-{Assembly.GetExecutingAssembly().GetName().Name.ToLower().Replace(".", "-")}-{hostEnvironment.EnvironmentName.ToLower().Replace(".", "-")}-logs-{DateTime.UtcNow:yyyy-MM}"; configuration
.Enrich.FromLogContext()
.Enrich.WithMachineName()
.Enrich.WithEnvironmentName()
.Enrich.WithEnvironmentUserName()
.Filter.ByExcluding(x => x.Level == LogEventLevel.Debug)
.WriteTo.Elasticsearch
(
new ElasticsearchSinkOptions(new Uri(hostContext.Configuration["ElasticConfiguration:Uri"]))
{
IndexFormat = IndexFormat,
AutoRegisterTemplate = true,
NumberOfShards = 2,
NumberOfReplicas = 1,
}
)
.Enrich.WithProperty("Enviroment", hostContext.HostingEnvironment.EnvironmentName)
.ReadFrom.Configuration(hostContext.Configuration);
};
}This is class named SeriLoggerUtility which defines a Configure method that is used to configure a Serilog logger.
The Configure method takes in two parameters: hostContext of type HostBuilderContext and configuration of type LoggerConfiguration.
Within the Configure method, a variable Environment is assigned the value of the ElasticConfiguration:Environment configuration value. Then, an index format string is constructed using various components such as the name of the entry assembly, the current environment name, and the current UTC date.
Next, the LoggerConfiguration instance is enriched with various properties such as the machine name, environment name, and environment user name. It also excludes log events with a Debug level and writes log events to an Elasticsearch sink using the constructed index format and some other options such as the number of shards and replicas. Finally, the logger configuration is enriched with an Enviroment property and is read from the provided hostContext configuration.
How to use:
After adding the library to our project, we have to follow the following steps:
1 — We will add some lines to Program.cs
using Distributed.Logging.Utilities;
var builder = WebApplication.CreateBuilder(args);
builder.Host.ConfigureAppConfiguration((hostingContext, config) =>
{
config.AddEnvironmentVariables();
}).UseSerilog(SeriLoggerUtility.Configure);
var app = builder.Build();
app.UseMiddleware<ExceptionHandlingMiddleware>();
app.Run();Here we will Sets Serilog as the logging provider and add ExceptionHandlerMiddleware to pipeline
2 — add Elastic URL in appseting.json .
{
"Serilog": {
"MinimumLevel": {
"Default": "Information",
"Override": {
"Microsoft": "Information",
"System": "Warning"
}
}
},
"ElasticConfiguration": {
"Uri": "http://localhost:9200"
}
}Like of the project in GitHub:
