Source: pixabay.com

Missing traces/exceptions in Application Insights

Introduction

Azure Application Insights is a PaaS solution by Microsoft that you can use to collect telemetry (requests, log traces, exceptions, etc.) from various sources and then query on this data. It integrates well with other Microsoft technologies (.NET applications and most Azure resources) but also offers open-source SDKs for other frameworks and programming languages.

The problem

I came across this twice, recently, where I have an application that sends telemetry to Application Insights but somehow not everything can be found by querying for it. I had a Function App that called logger.LogInformation() in order to log certain information. When querying for these traces, I noticed how some (not all) traces were missing. I did find an answer to this mystery and decided to write an article about it, which may help other Azure developers who run into this issue.

Troubleshooting

The first thing to do is verify if there is any telemetry at all in Application Insights:

1. Go to your Application Insights resource via portal.azure.com
2. Navigate to Monitor > Logs
3. Run this query:

union requests,dependencies,exceptions,traces
| where timestamp > ago(24h)
| order by timestamp desc

Note: the query above is in the Kusto Query Language (learn more).

If the query above doesn’t reveal any of the telemetry you expect then you probably misconfigured something; check if you have used the correct InstrumentationKey and if you are correctly initializing the SDK (if you use one).

If the query does show some of the telemetry you expect, but not all; run this query:

union requests,dependencies,pageViews,browserTimings,exceptions,traces
| where timestamp > ago(24h)
| summarize RetainedPercentage = 100/avg(itemCount) by bin(timestamp, 1h), itemType
| order by itemType, timestamp asc

This will reveal, per type of telemetry, how much is retained. If you see that RetainedPercentage for any type is less than 100, that indicates that it is being sampled.

Sampling

What is sampling you might wonder, let me quote Microsoft here:

Sampling is a feature in Azure Application Insights. It is the recommended way to reduce telemetry traffic, data costs, and storage costs, while preserving a statistically correct analysis of application data. Sampling also helps you avoid Application Insights throttling your telemetry.

Adaptive sampling is enabled by default in all the latest versions of the Application Insights ASP.NET and ASP.NET Core SDKs. It is also used by Azure Functions.

Source: https://docs.microsoft.com/en-us/azure/azure-monitor/app/sampling

Solution

You might think “So I will just disable sampling then.” but it is important to understand that sampling is a feature which can have significant benefits, especially in situations where you have a high throughput.

Before simply disabling sampling all together, take a moment to learn how it works and what the benefits are. Then consider using a more fine-grained approach by only applying adaptive sampling to specific types of telemetry.

For .NET applications, you can do so by excluding (or including) specific types of telemetry in your ApplicationInsights.config file:

<ExcludedTypes>Trace;Exception</ExcludedTypes>

Learn more about Configuring Adaptive Sampling in .NET or Configuring Adaptive Sampling in Azure Functions.