Pushing Your Logs to Datadog Through Serilog Sink

Integrating your .NET Core application with Serilog and Datadog, to have great visibility over what’s going on in your code.

Image by Randy ORourke on Unsplash

What is Serilog?

Serilog is a structured and portable logging framework to record diagnostic logs to files, the console, and elsewhere.

Structured logging improves the efficiency and scalability of searching and analyzing code by separating the event capture process from its transportation and presentation to the end-user. Instead of viewing logs as a series of text lines, they are treated as a set of key/value properties with timestamps. For instance, consider the difference between these two lines:

# text
2021-10-07 09:05:37 [INF] UsersController:GetUserDetails#45 getting details for user with ID 16
# structured
time=2021-10-07 09:05:37, level=INF, service=UsersController, method=GetUserDetails, line_num=45, content=details for user with ID 16

If we inspect the second log carefully, we’ll realize that it’s actually a JSON. So let's rewrite it as a one:

{
  "time": "2021-10-07 09:05:37",
  "level": "INF",
  "service": "UsersController",
  "method": "GetUserDetails"
  "line_num": 45,
  "content": "details for user with ID 16"
}

Tools like Datadog are there to helps us displaying, analyzing and searching structured logs in a very efficient manner.

Serilog facilitates the routing of logs to various destinations using a concept known as “sinks”. These sinks allow logs to be sent to a text file, database, or cloud-based monitoring platform such as Datadog. The illustration below provides a visual representation of how sinks work:

Image by codeproject

Integrating Serilog With .NET Core

First, install the Serilog.AspNetCore NuGet package into your app.

dotnet add package Serilog.AspNetCore

Next, let’s add a basic configuration to our app settings JSON:

{
  "Serilog": {
    "Using": [ "Serilog.Sinks.Console" ],
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "System": "Error",
        "AWSSDK": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "Console",
        "Args": {
          "formatter": "Serilog.Formatting.Compact.CompactJsonFormatter, Serilog.Formatting.Compact"
        }
      }
    ],
    "Enrich": [ "WithThreadId", "WithMachineName" ]
  },
  "Datadog": {
    "url": "https://http-intake.logs.datadoghq.eu"
  }
}

Don’t be concerned about the details now, we will provide a thorough explanation later.

Now, let’s add the Serilog sink for Datadog logs package:

dotnet add package Serilog-Sinks-Datadog-Logs

To integrate Serilog into our application, we use the UseSerilog method. In the Program.cs file, insert the following:

public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
    WebHost.CreateDefaultBuilder(args)
        .UseSerilog()
        .UseStartup<Startup>();

Now we will configure the Datadog sink in our Startup.cs file:

var datadogConf = new DatadogConfiguration(configuration.GetValue<string>("Datadog:url"));
logger = logger.WriteTo.DatadogLogs(
    _vaultService.GetSecret("DatadogApiKey"),
    service: "development-backend",
    host: Environment.GetEnvironmentVariable("DD_HOST") ?? "production",
    configuration: datadogConf
);

Datadog:url is the Datadog API URL (e.g. https://http-intake.logs.datadoghq.eu), while we get the API key from our vault.

In addition, we add the following enrichers to make our logs even more verbose:

using System.Runtime.CompilerServices;
using Serilog;

namespace company;

/// <summary>
/// This class is used to enrich the default Serilog fields.
/// For example, we can add the method's name, and the exact line number from
/// which the log is executed.
/// </summary>
public static class LoggerExtensions
{
  public static ILogger Enrich(this ILogger logger,
    [CallerMemberName] string memberName = "",
    [CallerLineNumber] int sourceLineNumber = 0)
  {
    return logger
      .ForContext("MemberName", memberName)
      .ForContext("LineNumber", sourceLineNumber);
  }
}

We use this enricher to be able to see the source line number and the member name in our logs.

Finally, we need to install the following packages:

SeriLog.Enrichers.Environment
Serilog.Enrichers.Thread
Serilog.Formatting.Compact

Using the Logger

Here’s an example of using the logger:

public class UserService : IUserService
{
    private static readonly ILogger Logger = Log.ForContext(MethodBase.GetCurrentMethod().DeclaringType).Enrich();
    
    public async Task<HttpResponseMessage> UpdateUserName(string userName)
    {
        Logger.Information("updating username {@userDetails}", 
            new { currentName = context.GetUserName(), newName = userName });
        // ...
    }
}

In Datadog, we should see something like:

Datadog

Datadog

All fields of the above JSON can be customized through the Datadog dashboard, and viewed in a separate column, making your important logs very verbose and customizable.

Summary

By utilizing the Serilog sink, you can publish structured logs to Datadog, providing a centralized platform to analyze, monitor, and search for events within your application. The ability to send logs to Datadog allows for improved visibility and control over your application, making it easier to identify and resolve issues in real-time. This results in increased reliability and performance for your application and a better user experience.