How Azure’s Application Insights correlates telemetry

A while ago I discovered that Azure Application insights can automatically stitch together the logs from the message consumer and message producer of Azure Service Bus messages, despite being saved in different Application Insights resources.

This has been a long standing pain point for our message based architecture. Now when we publish a message we have a sense of its outcome. Recently I showed this to a colleague. Impressed, he followed up with a predictable question: “how does it work?”

This post describes the output of hours digging through incomplete, out-of-date documentation and source code to answer this question.

The first thing we need to understand is how Application Insights knows about your application dependencies, such as SQL calls and HTTP requests. Naturally, the official docs provides little detail on how dependency collection works. However, they do leave us with one clue:

Monitoring works by using byte code instrumentation around select methods or based on DiagnosticSource callbacks (in the latest .NET SDKs) from the .NET Framework. Performance overhead is minimal.

Since we’re dealing with ASP.NET Core we can safely assume we’re using DiagnosticSource.

For instance, System.Net.Http.HttpClient will automatically add a DiagnosticsHandler that is responsible for publishing diagnostics about when outbound requests start and end. If a diagnostic listener is subscribed, it can access the entire HttpRequestMessage and HttpResponseMessage objects. This provides a powerful hook to log and enrich outbound messages and their responses.

Some more digging takes us to the DiagnosticSource User’s Guide. The salient point is that in dotnet has a feature called `DiagnosticSource` that allows libraries to publish messages such as instrumentation information that any other library can listen to and even modify. Application Insights can automatically track dependencies to supported targets since these libraries implement DiagnosticSource publishers . Using this hook, it can read messages, enrich them with extra data and log them.

Microsoft.ApplicationInsights.Web.DependencyCollector source code reveals that Application Insights uses these hooks to attach additional HTTP headers so that the target service can parse and correlate them to a parent operation.

Astute readers will notice, in typical Microsoft fashion, the code base is littered with deprecation and legacy comments. Further, there are no less than three formats for tracing headers in the code base: Legacy Headers, W3C Distributed Tracing Proposal and Microsoft's HTTP Correlation Protocol.

With this background knowledge on DiagnosticSource out of the way we can now proceed with an illustrated example of how everything works:

1. A user sends a httpRequest to a ‘my-api-service’. This is the root operation and contains an id.
2. Inside the context of that request it can call a few APIs.
3. The application publishes a service bus message. Just before publishing Azure’s Application Insights .NET SDK will hook into the event for publishing and add an extra header called Diagnostic-Id to the message. This value is a hierarchical id that has the dependency and root operation id encoded in it. The service bus message is also logged as a custom dimension. This is illustrated as a ‘queue’ event in ‘my-servicebus-namespace’.
4. Another service will consume the message. If application insights is installed on the consuming app, Microsoft.ApplicationInsights.Web.DependencyCollector will read the message and look for the Diagnostic-Id header value. The processing of the service bus operation is logged as a request telemetry, it uses the received Diagnostic-Id as its operation parent. This means that step 3 is a child of step 2.

With this technical report in hand, I return to my colleague; he ponders it for a moment… eventually he says excitedly ‘I get it now!’ as I watch the sense of understanding wash over him.