The state of monitoring in Azure
I’ve recently developed a strong interest for performance monitoring. This is the opportunity for me to expand my skills on Application Insights, which I’ve been using for many years, but also complement them with an overview of all the monitoring services available on Azure.
There are currently 3 different services related to monitoring on Azure:
As often on Azure, these services provide similar and overlapping features and even though this page gives helpful examples of “when to use which”, I thought I would outline here, and in my own words, a review of their features, differences and similarities.
Infrastructure vs. application monitoring
Infrastructure and application monitoring are very different tasks that require different metrics and different exploitation of those metrics.
Infrastructure monitoring calls for both a detailed reporting of the activity of each service of your system, and a synthetic, high-level overview of the global health of that system. It should also give the ability to conveniently overlay metrics from different services to facilitate the discovery of correlations. This can be achieved with both Monitor and Log Analytics, with differences explained in the next sections hereunder.
Application monitoring requires the collection of data much closer to the code and instruments your code in order to do that. It collects query performance, custom logs and thorough exception data, so you have all the information at hand to quickly identify the piece of code to investigate in case of issues. That’s what Application Insights’ SDKs and agents provide, with a basic instrumentation experience out of the box and the ability to customize it to make it more specific to your code.
Monitor provides infrastructure-level monitoring by collecting metrics from most of your Azure resources with near-to-zero configuration. Monitor is actually activated by default on each Azure subscription and doesn’t have to be provisioned. Also by default, it leverages Azure Diagnostics to automatically collect telemetry from all compatible resources that exist in your subscription.
From the collected telemetry, Monitor lets you:
- Visualize it on the Azure portal and build charts that you can pin on dashboards
- Store it in Azure Storage if you want to access it beyond the default retention limits and/or use it with external monitoring solutions
- Query it through a number of channels including a REST API, the Azure CLI and Powershell commands
- Trigger alerts and auto-scale actions based on custom rules
Log Analytics, which is a kind of spin-off from Operations Management Suite, is basically similar to Monitor in that it is suited to infrastructure monitoring, but can ingest telemetry from a wider range of sources (and not only Azure resources); it can
- Receive the same telemetry as Monitor (through Azure Diagnostics)
- Collect additional metrics and logs through connectors
- Load external telemetry from Azure Storage
From the collected telemetry, Log Analytics lets you:
- Search interactively on the Azure portal
- Create custom visualizations and build elaborate dashboards
- Trigger alerts based on custom rules
- Use “Solutions”, which I would describe as templates that provide off-the-shelf dashboards and alerting for typical resources (like Web Apps or SQL Server instances for example)
- Access a separate “advanced analytics” portal where you can execute custom queries over the telemetry data, run machine learning algorithms and generate charts that you can pin on your dashboards
Application Insights is a complete application performance management product that lets you monitor the performance of your application (most likely your web app, although it can be used to instrument and gather custom telemetry from any kind of app). It is mainly composed of:
- A service hosted on Azure that receives telemetry from agents
- A comprehensive collection of interactive visualizations and analytics widgets available on the Azure portal that help you monitor the telemetry and troubleshoot issues (check this post to see how to explore performance issues with Application Insights!)
- The very same “advanced analytics” portal as available on Log Analytics (providing complex querying, machine learning and charting)
- A handy integration with Visual Studio to access and explore performance data directly from the IDE
The big picture
None of these 3 products would allow you to achieve both infrastructure and application monitoring in a satisfying form and in most cases, you will need to use both Application Insights and (at least) one of Monitor or Log Analytics. One exception though would be if you have a simple Web App with minimum infrastructure around it; in that case, the performance counter collection that Application Insights does on the underlying App Service may be enough for you.
Monitor or Log Analytics?
As explained above, these 2 services have similar goals and features so let’s try to outline some key considerations to take into account when deciding which one to use.
If all your infrastructure is within Azure, you should start with Monitor. It doesn’t require any kind of setup and enables you to quickly assemble charts and pin them to your dashboards. If you know that some of your services will be sitting outside of Azure, like VMs running on-premises or in other clouds, it may be wiser to invest in Log Analytics from the start.
I just said invest in Log Analytics because its learning curve is steeper than with Monitor. Except for some Azure resources, collection of metrics isn’t activated with just the click of a button and may require some Powershell commands or the configuration of agents. Similarly, construction of charts is less straightforward and requires familiarity with the query language.
Another way to differentiate these 2 services is by their query capabilities. Monitor offers limited querying (I should rather call it “filtering”) whereas Log Analytics gives you access to the full power of the Kusto query language through the separate, “advanced analytics” portal. In other words, Monitor is fine for out-of-the-box visualizations but won’t let you investigate the data you have collected.
Get started with monitoring on Azure!
A recommended path for those who are not sure where to start:
- Begin with Monitor and Application Insights. Both are easy to setup (you add the Application Insights agent to your ASP.NET app with just a couple of clicks from Visual Studio) and offer rich charting capabilities
- Create a monitoring dashboard on your Azure portal and pin some key charts on it
- Take some time to learn the Kusto query language that powers the “advanced analytics” sections of both Log Analytics and Application Insights; it’s incredibly powerful to explore data and create custom visualizations — and it’s pretty fun to use!
- If/when you feel that you’ve reached the limits of what Monitor can provide and need to dig deeper in your infrastructure telemetry, consider switching to Log Analytics