Integrating Kibana Logging in a .NET Application

3 min read6 days ago

Logging is a crucial aspect of any application, providing insights into its operation and aiding in debugging. Kibana, combined with Elasticsearch and Logstash (collectively known as the ELK stack), offers a powerful solution for managing and visualizing logs. This blog will guide you through integrating Kibana logging in a .NET application using Serilog and Elasticsearch.

Embark on a journey of continuous learning and exploration with DotNet-FullStack-Dev. Uncover more by visiting our https://dotnet-fullstack-dev.blogspot.com reach out for further information.

What is Kibana?

Kibana is a data visualization and exploration tool used for log and time-series analytics, application monitoring, and operational intelligence use cases. It allows you to visualize Elasticsearch data and navigate the Elastic Stack, providing powerful search and data aggregation capabilities.

Why Integrate Kibana Logging?

Integrating Kibana logging in your .NET application offers several benefits:

Centralized Logging: Aggregate logs from multiple sources into a single location.
Real-time Monitoring: Monitor application health and performance in real-time.
Advanced Search and Analysis: Utilize Kibana’s powerful search and filtering capabilities.
Visualization: Create dashboards to visualize logs and metrics.

Prerequisites

.NET SDK installed.
ELK stack set up (Elasticsearch, Logstash, Kibana).
Basic understanding of logging in .NET.

Step-by-Step Integration

Step 1: Set Up ELK Stack

Ensure that you have the ELK stack set up and running. You can follow the official Elastic documentation for installation and setup.

Step 2: Create a .NET Application

Create a new .NET application (e.g., ASP.NET Core Web API).

dotnet new webapi -n KibanaLoggingDemo
cd KibanaLoggingDemo

Step 3: Add Serilog and Serilog.Sinks.Elasticsearch

Add the necessary packages to your project.

dotnet add package Serilog.AspNetCore
dotnet add package Serilog.Sinks.Elasticsearch

Step 4: Configure Serilog

Configure Serilog to log to Elasticsearch in the Program.cs file.

using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Hosting;
using Serilog;
using Serilog.Formatting.Compact;

public class Program
{
    public static void Main(string[] args)
    {
        Log.Logger = new LoggerConfiguration()
            .Enrich.FromLogContext()
            .WriteTo.Console(new CompactJsonFormatter())
            .WriteTo.Elasticsearch(new Serilog.Sinks.Elasticsearch.ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
            {
                AutoRegisterTemplate = true,
            })
            .CreateLogger();

        try
        {
            Log.Information("Starting web host");
            CreateHostBuilder(args).Build().Run();
        }
        catch (Exception ex)
        {
            Log.Fatal(ex, "Host terminated unexpectedly");
        }
        finally
        {
            Log.CloseAndFlush();
        }
    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .UseSerilog()
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseStartup<Startup>();
            });
}

In .Net 6 when you don’t have Startup and Main classes it can be used in program.cs

public class Program
{
    public static void Main(string[] args)
    {
        
        try
        {
            var builder = WebApplication.CreateBuilder(args);
            var logger = new LoggerConfiguration()
                                .ReadFrom.Configuration(builder.Configuration)
                                .Enrich.FromLogContext()
                                .WriteTo.Console(new CompactJsonFormatter())
                                .WriteTo.Elasticsearch(new Serilog.Sinks.Elasticsearch.ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
                                {
                                    AutoRegisterTemplate = true,
                                })
                                .CreateLogger();
            builder.Host.UseSerilog(logger);
            builder.Services.AddControllers();
            // Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
            builder.Services.AddEndpointsApiExplorer();
            builder.Services.AddSwaggerGen();

            // more configuration

            var app = builder.Build();
           
            // Configure the HTTP request pipeline.
            if (app.Environment.IsDevelopment())
            {
                app.UseSwagger();
                app.UseSwaggerUI();
            }
            app.UseSerilogRequestLogging();
            app.UseHttpsRedirection();

            app.UseAuthorization();


            app.MapControllers();
            app.UseSerilogRequestLogging();

            // more configs
            app.Run();
        }
        catch (Exception ex)
        {
            Log.Fatal(ex, "Unhandled exception");
        }
        finally
        {
            Log.Information("Shut down complete");
            Log.CloseAndFlush();
        }

    }
}

Step 5: Configure Logging in `appsettings.json`

You can add additional configuration for Serilog in the appsettings.json file.

{
  "Serilog": {
    "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.Elasticsearch" ],
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "System": "Warning"
      }
    },
    "WriteTo": [
      { "Name": "Console" },
      {
        "Name": "Elasticsearch",
        "Args": {
          "nodeUris": "http://localhost:9200",
          "indexFormat": "logstash-{0:yyyy.MM.dd}",
          "autoRegisterTemplate": true,
          "templateName": "serilog-events-template"
        }
      }
    ]
  }
}

Step 6: Use Logging in Your Application

Use the configured logger in your application. For example, in a controller:

using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;

namespace KibanaLoggingDemo.Controllers
{
    [ApiController]
    [Route("[controller]")]
    public class WeatherForecastController : ControllerBase
    {
        private static readonly string[] Summaries = new[]
        {
            "Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
        };

        private readonly ILogger<WeatherForecastController> _logger;

        public WeatherForecastController(ILogger<WeatherForecastController> logger)
        {
            _logger = logger;
        }

        [HttpGet]
        public IEnumerable<WeatherForecast> Get()
        {
            _logger.LogInformation("Getting weather forecast");

            var rng = new Random();
            return Enumerable.Range(1, 5).Select(index => new WeatherForecast
            {
                Date = DateTime.Now.AddDays(index),
                TemperatureC = rng.Next(-20, 55),
                Summary = Summaries[rng.Next(Summaries.Length)]
            })
            .ToArray();
        }
    }
}

Step 7: Visualize Logs in Kibana

Navigate to Kibana (usually at http://localhost:5601) and create an index pattern that matches the indices created by Serilog (e.g., logstash-*). You can then visualize your logs in Kibana using various dashboards and tools.

Conclusion

Integrating Kibana logging in a .NET application using Serilog provides powerful logging capabilities, including centralized log management, real-time monitoring, advanced search, and data visualization. This guide provided a step-by-step approach to setting up and configuring logging to Elasticsearch and visualizing logs in Kibana. By following these steps, you can gain valuable insights into your application’s operation and performance.