Instrumenting .NET chiseled Docker images with Datadog

Raphaël Vandon
5 min readJun 24, 2024

--

Microsoft recently announced the availability of “chiseled” containers images that can be used to run dotnet applications. Those are images that are stripped to the bare minimum.

The most obvious benefit from using a chiseled base is that the produced image is smaller in size, but it also helps on the security front, in part because those images pack no shell. Which means it’s impossible to run commands inside the container. This makes it a lot harder for malicious actors to do anything if they gain access, but it also restricts what we can do when we setup the image.

The documented way to instrument a dotnet app in Docker with Datadog is to install it using the package manager, and then run a script that’ll create a log folder. Two things that are not possible with a chiseled image. Here I’ll explain how we can achieve the same goals without running commands.

Installing the Datadog tracer

This is the easiest part, We’re just going to prepare the files in a separate stage (where we have a shell, for instance an alpine:latest image), and then copy them in the chiseled image. I’m downloading the tar.gz using the Docker ADD command.

FROM alpine:latest AS dd-tracer-stage
ADD https://github.com/DataDog/dd-trace-dotnet/releases/download/v2.53.2/datadog-dotnet-apm-2.53.2.arm64.tar.gz .
RUN mkdir extracted-tracer
RUN tar -C /extracted-tracer -xzf datadog-dotnet-apm-*.tar.gz

FROM mcr.microsoft.com/dotnet/runtime:8.0-jammy-chiseled-extra
COPY --from=dd-tracer-stage /extracted-tracer/ /opt/datadog/

Here I’m using the version 2.53.2 because it’s the latest at the time of writing, and I’m using the arm64 flavor because I’m building an image I’ll run locally on my Mac. I put the content in /opt/datadog because it’s the standard location.

Preparing the logs folder

This one is a little bit harder, because we cannot use mkdir (no shell, remember?). The trick we’ll use here is to create an empty folder first (in a different stage of the build for instance), and then we’ll copy this empty folder where we need one:

FROM alpine:latest AS dd-tracer-stage
RUN mkdir -p /empty

FROM mcr.microsoft.com/dotnet/runtime:8.0-jammy-chiseled-extra
COPY --chown=$APP_UID --from=dd-tracer-stage /empty/ /var/log/datadog/dotnet/

It’s really important to use --chown=$APP_UID here, otherwise, the user running the app won’t have permissions to write the logs to that folder (it’d be owned by root). Note the trailing / on the src and dest path, so that the content of the folder (here, nothing) is copied, and not the folder itself.

Putting it all together

This is what the files look like in my sample project:

The Program.cs is very basic, and looks like this:

using Datadog.Trace.Annotations;

DoWork();
// Wait so that the container stays up and can be inspected
Thread.Sleep(TimeSpan.FromHours(1));

[Trace(ResourceName = "DoWork")] // Create a span for this function
static void DoWork()
{
Thread.Sleep(1000);
Console.WriteLine("Work done.");
}

I’m specifying the resource manually on the Trace annotation here, because on top-level-defined functions like this one, the resource name that is automatically generated is Program.<<Main>$>g__DoWork|0_0 (which doesn’t look great).

This is the full Dockerfile:

# Using a very minimal image to prepare the tracer
FROM alpine:latest AS dd-tracer-stage
ADD https://github.com/DataDog/dd-trace-dotnet/releases/download/v2.53.2/datadog-dotnet-apm-2.53.2.arm64.tar.gz .
RUN mkdir extracted-tracer
RUN tar -C /extracted-tracer -xzf datadog-dotnet-apm-*.tar.gz
# Create an empty folder we can copy in the next stage to create the logs folder
RUN mkdir -p /empty

# Building the code like we would usually
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build-stage
WORKDIR /App
COPY . ./
RUN dotnet restore
RUN dotnet publish -c Release -o out

# Runtime image (chiseled)
FROM mcr.microsoft.com/dotnet/runtime:8.0-jammy-chiseled-extra
WORKDIR /App
# Copy the published app
COPY --from=build-stage /App/out .
# "Install" the Datadog tracer
COPY --from=dd-tracer-stage /extracted-tracer/ /opt/datadog/
# Create folder for logs
COPY --chown=$APP_UID --from=dd-tracer-stage /empty/ /var/log/datadog/dotnet/

# Set environment variables
ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}
ENV CORECLR_PROFILER_PATH=/opt/datadog/Datadog.Trace.ClrProfiler.Native.so
ENV DD_DOTNET_TRACER_HOME=/opt/datadog
ENV DD_TRACE_DEBUG=1
ENV DD_SERVICE=sample-app

# Start the application (will run as $APP_UID)
ENTRYPOINT ["dotnet", "Demo.dll"]

And the docker-compose is nothing original, but I’m putting it here anyway for good measure (it contains the Datadog agent, setup as described in the doc, and my sample app):

services:
dd-agent:
container_name: dd-agent
image: "gcr.io/datadoghq/agent:latest"
environment:
- DD_API_KEY=${DD_API_KEY}
- DD_SITE=${DD_SITE} # =datadoghq.eu in my case
- DD_ENV=dev # Change as necessary
ports:
- "127.0.0.1:9126:8126" # I'm mapping the agent's port 8126 to a different one locally, because I also run a local agent.
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /proc/:/host/proc/:ro
- /sys/fs/cgroup/:/host/sys/fs/cgroup:ro

dotnetapp:
build: .
environment:
- DD_AGENT_HOST=dd-agent

And with all that, if I run docker-compose up -d --build, I can see the trace being produced in the Datadog UI 🎉

Looking at the tracer logs

Since there is no shell in the image we’re using, we cannot docker exec into it to take a look. There are alternatives though. When running with docker desktop, the UI offers a way to quickly take a look at the file system, and we can then open the file editor to look at the logs.

This solution is only useful for development purposes though, because you won’t have docker desktop on a server. The command line solution is to use docker cp to copy the logs locally, where they can be observed:

$ docker ps
CONTAINER ID IMAGE COMMAND ...
f143b95eb72c gcr.io/datadoghq/agent:latest "/bin/entrypoint.sh" ...
21e4ebb73ec4 demo-dotnetapp "dotnet Demo.dll" ...

$ docker cp 21e4ebb73ec4:/var/log/datadog/dotnet/ ./logs
Successfully copied ...kB to ...

You now know everything you need to trace your .NET apps running on chiseled images using Datadog!

--

--