Logging and monitoring the dotnet API using serilog and Azure Application Insights

Introduction

In the dynamic world of software development, creating robust and reliable applications is crucial to meet the ever-changing needs of users and businesses. As the complexity of .NET Core APIs grows, the need for effective logging and monitoring becomes indispensable.

Logging

Logging is the process of gathering crucial data about the actions taken by your program while it is running. It aids in the development of insights into the application’s execution flow, the detection of faults, and the diagnosis of potential problems. When logging is used correctly, it can speed up debugging and troubleshooting, giving developers more time to address any issues.

Prerequisites

• .Net
• C#

In this article I will assume that you know how to create a dotnet webapi or you already have created one.

Application Insights

Microsoft Azure offers a service called Application Insights that aids in monitoring the application. We can also use it as a central repository for keeping the logs we produce with Serilog or log4net.

First, we will need to install nuget packages for application insights. There are two ways to do this:

First Method

• Adding appinsight from visual studio Right click on the project:

• then, click on Add > Application Insights Telemetry that will open the below popup:

• We can directly connect it to azure app insight but we will do that manually. For now select the first option Application Insights Sdk (Local).
• Now, another popup will open as below:

• It is a confirmation screen warning that a nuget packages will be added and also some code will be added, just click Finish.
• The above process will add a nuget package Microsoft.ApplicationInsights.AspNetCore and also adds below code in Program.cs.

builder.Services.AddApplicationInsightsTelemetry();

Second Method

• This is much simple, we go to the nuget package manager and search for the package Microsoft.ApplicationInsights.AspNetCore and install it in the project as shown in the below image.

• Then we manually add the code to the file Program.cs as below:

builder.Services.AddApplicationInsightsTelemetry();

Now, after this our app will be monitored continously. To see that run the application and try to call APIs, run functions, etc.

To check the insights go to Visual studio and on the top right corner you will see application insights like below:

Note: If you dont see the icon click on the down arrow and it will be as a dropdown.

Now, you will be able to see all the details just as below:

Next we will configure Serilog to generate the logs. We will need to install the below nuget packages:

• Serilog
• Serilog.AspNetCore
• Serilog.Sinks.File

After installing the packages, we will need to configure serilog which also has few methods but I will only show one method in this article which I like the most.

In appsetting.json file we add the below section:

{
  "Serilog": {
      "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.File" ],
      "MinimumLevel": {
         "Default": "Information",
         "Override": {
            "Microsoft": "Warning",
            "System": "Warning"
         }
      },
      "WriteTo": [
         { "Name": "Console" },
         {
            "Name": "File",
            "Args": {
               "path": "logs/log-.log",
               "rollingInterval": "Day",
               "rollOnFileSizeLimit": true
            }
         }
      ]
   }
}

In the above json Using defines the packages that we import just like when we use using in C# file. Then there is some log information and WriteTo defines where the logs will be written, in this case first on console and then in a file which will be named as logs/log_{current_date}.log where the current date is the date in format YYYYMMDD.

In the Program.cs we will add the serilog configuration:

builder.Host.UseSerilog((context, configuration) =>
    configuration.ReadFrom.Configuration(context.Configuration));

then, add the serilog middleware above the app.Run();

app.UseSerilogRequestLogging();

Now we can inject logger in any C# class and log some details as shown in the below example:

public class DepartmentController : ControllerBase {
  private readonly ILogger<DepartmentService> _logger;
  public DepartmentController(ILogger<DepartmentService> logger) {
      _logger = logger;
  }

  public IActionResult LogSomething() {
    _logger.LogInformation("Logging is working just fine.");
  }
}

To see these logs in Application Insights we will need to modify the serilog settings in the appsettings.json. Install the nuget package Serilog.Sinks.ApplicationInsights.

Then, change the serilog > WriteTo section in appsettings.json and add application insights part as below:

{
   "Serilog": {
      "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.File", "Serilog.Sinks.ApplicationInsights" ],
      "MinimumLevel": {
         "Default": "Information",
         "Override": {
            "Microsoft": "Warning",
            "System": "Warning"
         }
      },
      "WriteTo": [
         { "Name": "Console" },
         {
            "Name": "File",
            "Args": {
               "path": "logs/log-.log",
               "rollingInterval": "Day",
               "rollOnFileSizeLimit": true
            }
         },
         {
            "Name": "ApplicationInsights", // Write Logs to App Insights
            "Args": {
               "telemetryConverter": "Serilog.Sinks.ApplicationInsights.TelemetryConverters.TraceTelemetryConverter, Serilog.Sinks.ApplicationInsights"
            }
         }
      ]
   },
}

The above code will show all the logs in application insights including the ones that are logged using serilog. Example, the below logging code:

_logger.LogInformation("Fetching departments: {departments}", departmentDto);

The many filters that we can use to examine various types of logs are indicated in red in the image above. The serilog logs are typically written under trace. The logs we wrote are indicated in blue. You will get all the details about a specific log by clicking on it.

After all the above process, now we want these logs to be shown in application insights service in azure portal. For that we will create a service first by follwing the next few steps.

Create an azure account or if you already have then create a service application insights by following these steps. Goto azure portal:

As in the image the second option is Application Insights, if you don’t see the application insights there then you can search it. Next you click on that and the application insights service screen will popup.

Click on Create from top left corner or just click the center button Create Application Insights apps and it will take to the below screen:

Select the subscription and Resource Group (If there is no resource group to select then click on create new and enter the name of the resource group it will create one). Then, enter name of you service e.g: log-appinsight-demo in my case. Then select region and click Review + Create.

Once the validation is passed you can now click on Create button below.

After taking some time to deploy the service you will get the below screen:

Click on Go to resource and it will take you to overview screen of the created app insight service:

Now we will need the Connection String from this screen as we can see, I have highlighted it with red line. Hover over to it copy the connection string.

Finally, in you .net aplication open appsettings.json file and write the below line in the section Serilog > WriteTo > "Name": "ApplicationInsights"

"connectionString": "the_copied_connection_string_goes_here"

The serilog section will be as the following code:

"Serilog": {
   "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.File", "Serilog.Sinks.ApplicationInsights" ],
   "MinimumLevel": {
      "Default": "Information",
      "Override": {
         "Microsoft": "Warning",
         "System": "Warning"
      }
   },
   "WriteTo": [
      { "Name": "Console" },
      {
         "Name": "File",
         "Args": {
            "path": "logs/log-.log",
            "rollingInterval": "Day",
            "rollOnFileSizeLimit": true
         }
      },
      {
         "Name": "ApplicationInsights", // Write Logs to App Insights
         "Args": {
            "connectionString": "InstrumentationKey=abc_some_connection_string",
            "telemetryConverter": "Serilog.Sinks.ApplicationInsights.TelemetryConverters.TraceTelemetryConverter, Serilog.Sinks.ApplicationInsights"
         }
      }
   ]
}

To check the logs simply go to the azure portal open the application insights service and find Transaction Search in the side bar:

Once clicked in Transaction Search the following screen will popup:

Now, click on Refresh marked with red or just click See all data in the last 24 hours.

Now you can see the logs just like we saw in the visual studio above in this article.

Click on a single log and you will see all the details regarding it.

Conclusion

This brief demonstration showed you how to monitor your application and log activity in the Azure Application Insight Service. You may now explore more about how to query a certain log, how to programmatically get these logs and display them on the front end of your application, and many other things.

Thank you!