Structured Logging and Logs Management. A case study using ASP.NET Core, Serilog & Seq

9 min readNov 15, 2020

Nowadays, even the simplest application is logging data in one way or another, but what should be the best practice for writing your logs in order to get the most out of it, without spending too much of your precious time (or money).

This article provides a short background for the structured logging practice and walks you through a case study step by step to enabling structured logging in your application, managed by a fancy dashboard.

What do we want from our logs?

Well, that may vary from app to app, or between different companies, but here are 5 common things most of us wish to get from our logs:

Help to troubleshoot and diagnose issues in the application as quickly as possible
Audit significant activities and events in the application
Monitor system usage for preventing potential issues before they occur
Analyze system usage for improving the application usability
Feed other data analysis tools for supporting data-driven decision making

How structured logging can help with this?

Structured logging is the practice of treating logs as data that we are going to query and analyze. To support this, the application’s log is designed to be of consistent structure from the early development stage, with defined properties for your points of interest according to your application’s needs.

Most programming languages have libraries to support structured logging, which makes the implementation smoother and quicker. Accordingly, many great tools are integrated with these libraries to providing a log management interface. These tools are taking full advantage of the structured logging — digest, and serve the logs in a human-readable interface that makes the troubleshooting task easier. Most tools also let you set significant events (based on those properties you defined) to audit and monitor as well as sending alerts when needed. In this article, we will see such library and log management tool.

As time goes by, you will learn more about your app and your users, you’ll track potential bottlenecks, and get data-driven insights that would help you get better decisions for the next versions/upgrades, etc.

A case study using ASP.NET Core (3.1), Serilog & Seq — step by step

.NET Core has its built-in logging API that can be used as-is, but can also integrate with third-party logging frameworks neatly. You can find more information on the default logging in this detailed Microsoft docs article.

In this case, I used Serilog for my structured logging framework and Seq as my log management tool. There are many other options out there.

Configuring Serilog in your application for implementing structured logging

Serilog is a .NET Core third-party logging open-source library. It is simple to set up in your application and override the built-in Microsoft logging, it is structured logging oriented, has a large community of users, and relatively good documentation.

Serilog Sinks — Beyond all the above advantages of this library, it supports a very large range of so-called “Sinks” — ready to use integrations for writing your log events to other tools/platforms/storage. How large is that range? As of writing these lines, there are more than 90 available sinks, you may find the full list in this Provided Sinks article. One of these sinks is for the Seq log management tool we will use.

Serilog Custom Properties — When logging, there will be some predefined properties that are part of the structure and would be populated automatically. One of the real strengths of Serilog is in its ease of defining custom properties in your code. Following the required message template syntax, you can simply log different properties, depends on your specific function or feature. Locate your custom property name within {} brackets in the message string and pass the values as parameters. Below you will see several examples.

Steps for configuring Serilog -

The first step will be to add some Serilog packages to our project. In my case, I have added the below ones.

/*csproj file*/

<PackageReference Include="Serilog.AspNetCore" Version="3.4.0" />
<PackageReference Include="Serilog.Enrichers.Environment" Version="2.1.3" />
<PackageReference Include="Serilog.Enrichers.Process" Version="2.0.1" />
<PackageReference Include="Serilog.Enrichers.Thread" Version="3.1.0" />
<PackageReference Include="Serilog.Settings.Configuration" Version="3.2.0-dev-00244" />
<PackageReference Include="Serilog.Sinks.Seq" Version="5.0.0-dev-00174" />

Now, moving to our app.settings JSON file, we will delete the default Microsoft logging settings and set our own Serilog configurations including the Microsoft override, the Seq sink configurations, and some further enrichers and properties (which are optional).

/*app.settings (JSON file)*/

"Serilog": {
    "Using": [],
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "System": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "Seq",
        "Args": { "serverUrl": "MY_SEQ_SERVER_URL" }
      }
    ],
    "Enrich": [
      "FromLogContext",
      "WithMachineName",
      "WithProcessId",
      "WithThreadId"
    ],
    "Properties": {
      "ApplicationName": "MY_APP_NAME"
    }
  }

Next, we will need to modify our project Program.cs file in order to use Serilog as our application default logger and initialize the Serilog global logger instance with the configurations from our app.settings file.

/*PROGRAM.CS*/

//Main function

var config = new ConfigurationBuilder()
	.AddJsonFile("appsettings.json")
	.Build();

Log.Logger = new LoggerConfiguration()
	.ReadFrom.Configuration(config)
	.CreateLogger();

CreateHostBuilder(args).Build().Run();
	

//CreateHostBuilder function

Host.CreateDefaultBuilder(args)
                .UseSerilog() //<-Add this row
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();

                });

We are done. Now, in each class we would like to use our logger, all we need to do is to inject the Microsoft ILogger interface class (using dependency injection) and use its API functions. As we already set up the default logger override, Serilog will be used. There are several log levels, most used ones are Information, Warning, Error, and Critical. Below is an example of dependency injection and error logging.

/*Class (.cs file)*/

private readonly ILogger log;
//Constructor
public CLASS_NAME(ILogger<CLASS_NAME> logger)
        {
            log = logger;
        }
		
private void Func()
{

    log.LogError("The user {Username} could not authenticate." +
                    " Error message: {Error}", Username, ExceptionMsg);
}

Below is another example of attribute injection in a .NET Core Blazor app, logging Information log level.

/*Blazor component base class (.cs file)*/

[Inject]
ILogger<CLASS_NAME> log { get; set; }

private void Func()
{

    log.LogInformation("Resource was loaded for user {Username}", username);
}

Configuring Seq for log management & visualization

Seq is a free (single user) log management tool. Below is the company’s self-introduction.

Seq creates the visibility you need to quickly identify and diagnose problems in complex applications and microservices.

I tend to agree and think that it can help to improve your app beyond the troubleshooting tasks. Here are some of its great features -

Seq Stream — Seq is fully utilizing the structured logging, it provides a stream of all the logs with clear indications of different log levels. Once expanding a specific event, you get the mapping of all the properties and in 2 more clicks, you already query by property the whole stream. You can also filter and write your own SQL queries on top of the stream. It couldn’t be more intuitive.

Seq Dashboards — Who doesn’t like fancy dashboards? This is probably Seq’s biggest strength. It is very visible and easy to customize for your needs by adding any valuable stream filter/query in one click directly to your dashboard. That lets you audit your app’s significant events and monitor the usage (for example — in the below screenshot I set the “New Loans” widget to follow new book loan events in my library app). Clicking on any value in the dashboard immediately drills down to the relevant stream detailed results.

You can set the dashboard time range from 5 minutes and up to 1 year.. and set an automatic refresh to the screen. It comes with 6 different graph types. The widgets locations, sizes, colors are all customizable.

Seq Apps — Apps are plug-ins that process events and alerts or ingest events from an external source. There are currently around 70 available plug-ins, some of them are really useful and if you didn’t find the right one, you can write your own plug-in with their .NET package. You may check the list of available plug-ins. I will demonstrate how to use one of them for implementing alerts at the end of this article.

Steps for configuring Seq -

Seq gets its logs from several logging libraries that implemented this integration, that is available in different languages such as .NET, Python, Node.js, Java, Ruby, and Go, and it also supports using the HTTP ingestion API. In our case, the Serilog library works perfectly with Seq.

As you saw earlier, we already configured the Seq sink in our app.settings JSON file. So yes, with these 2 lines of code, all we need is to download & deploy our instance of Seq and it will work. It is that simple. Don’t forget to update the app.settings with the server URL of your Seq instance.

As for the tool installation, according to their great documentation, it currently supports deployment on Windows, Docker, Kubernetes, and Azure. Choosing your preferred deployment and following their instructions you should be good to go.

Now, you may run your application and refresh the Seq stream, query it, build your personal dashboards and start to configure the plug-ins you need for archiving, setting alerts and integrating with your company’s platforms. Later, add more of your applications to your new Seq tool and create different dashboards to manage all your applications logs from one place.

Bonus — Integrating your Seq logs with your Microsoft Teams

In this last chapter, I will demonstrate how to configure the MS Teams plug-in for Seq and set up alerts directly from the Seq dashboard to this popular communication platform. In this use case — an alert would be sent to my MS Teams for each log event of Error level.

Steps for configuring Seq-MS Teams Integration -

Go to Settings and select the APPS tab from the left navigation bar. Under the Package id, you will find a list of popular apps, one of them is Microsoft Teams — click on it and Install.

After the installation was completed, click on “Add Instance”. There are 2 mandatory fields — title and Teams WebHook URL. Open your MS Teams and set up a custom incoming webhook following the instructions in this Microsoft docs article. Copy the unique webhook URL and paste it in the Seq settings relevant field. In my case, I configured it under a “Log” team in my MS Teams. Once configured you should see the “Incoming Webhook” under the configured Connectors menu.

Navigate to your Seq dashboard. One of the widgets you get out of the box is the “Errors and Exceptions”. Expand it, and select the ALERTS tab, click the add icon, fill in the settings form and save the dashboard changes. Below is an example for settings that will send any error level log on intervals of 15 minutes to MS Teams.

That’s all. Your error level logs would now be sent to your MS Teams. Configuring alerts to your company’s communication platform is a great way to get notified on critical issues faster. There are many other available apps for integrating with your email server, Slack, Azure DevOps, Trello, and more.

The End.

Stav Sofer, November 2020