Setting up Alerts using Application Insights (the new ones)
In case you didn’t know, you can set up Alerts in Application Insights. However, what you could do with them was very, very (seriously, very) limited. Until now.
Just to give you an example, here is a list of things you COULD NOT do with the classic Alerts:
- Base your alerts on automated custom queries that return results
- In order to base your alert on a custom metric, you had to either send this metric several times to App Insights in order to make it visible in the Metric dropdown or provision the alert with one of the provided ways
- Use the ‘Equals to’ condition
- Customise the template of the email the alert sent is using
- Set the alert to specific action group
- Provide custom info about the problem in the email sent to make troubleshooting instant.
The biggest problem, in my opinion, is that unless you ping a custom metric for both success and failure all the time, you cannot use this system as it’s not automated on a scheduler.
However, this would work for any of the metrics that come out of the box.
Now let’s see what’s up with the new Alerts they recently released.
By the looks of it Microsoft took all that feedback and delivered.
The new alerts are located right above the old ones. It is easy to miss as their icon is exactly the same as the old one — but the old one has the word ‘classic’ in brackets next to it.
Once you click there, and you select ‘Create new alert’, the new workflow in order to create a new alert appears.
It is broken down into three sections:
- Define alert condition
- Define alert details
- Define action group.
Define alert condition
In this section you need to configure two things. The first thing that needs to be configured is the Application Insights instance that you are targeting. By default, this is set to the one you clicked the Alert button from.
The next one is where the magic happens. By clicking ‘Add criteria’ you open the ‘Configure signal logic’ menu.
Under Monitor Service there is a service called ‘Application Insights (Preview)’ which contains a single entry under it (as of 5 April 2018) called ‘Custom log search’.
Let’s see what the new view for setting up the new Custom log search looks like…
OK, let me explain what’s going on from top to bottom here.
The graph
The first thing you can see is a graph that currently displays no data.
This graph will display more info based on the Query, Period and Frequency. It is a visualisation tool based on the provided criteria. I found this very handy because I’m able to see the results of my query without having to run it in App Insight metrics.
Search query
This is the most important part. In there, you can paste the query you want app insights to run on schedule interval and check for results.
This is amazing but it comes with a limitation. You are not able to use things like the app("") function to cross query different app insights. I can live with that.
What this does behind the scenes is that it will append | count to the query provided to get the count of the results that this query returns.
Keep in mind that you should not add any timestamp related querying in the query you provided as this is something that you configure later in this view. In case it's there it will be ignored.
Alert logic
Pretty simple. Based on the number of results that this query returns you can set if the alert will be triggered or not.
Evaluated based on
Here is where you configure the period and the frequency of the check.
The limitation here is that Period cannot be less than Frequency (which makes perfect sense).
If you config configure Period and Frequency to 5 minutes then this query will be run against app insights every 5 minutes and it will check for results in the past 5 minutes.
NOTE: Because the evaluation is done every 5 minutes and metrics can take up to 2 minutes to show on application insights you alert might actually come from 7–10 minutes after the error occurred.
Define alert details
In this part you need to set up things like the name, the description the severity and whether you want this rule to be enabled upon creation. Simple stuff.
Define action group
Here is where you can select and action group or create one.
This essentially is about who will see this alert and has to take some action.
You can set it up to take many actions like: email, SMS, webhooks etc.
You can read more about action groups here: https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-action-groups
And that’s it. You are up and running.
The alert
What’s light years better than the previous alerting is that now the alert email actually includes useful data.
Here’s an example:
(A LOT of data underneath as well)
In comparison, that’s what the old alerts looked like.
Provisioning with ARM Template
So I exported the ARM template for this new type of alerts and I parameterised it.
You can thank me later. Or don’t. :sademoji:
Alert.json
Alert.paramaters.json
Common gotchas
Something I noticed (and it took me more time than I’d like to admit) is what I call the ‘Inactive Application Insights’ state.
You see, if you are provisioning Applications Insights right before you provision this alert then the deployment will fail and the error that the step will give back is that the query you have is invalid. Everything I write is amazing and there is no way I did something wrong (OK, I’ll show myself out) so I started investigating.
Long story short, Application insights has 2 different states.
Right after it’s provisioned, it has no data so it’s inactive until some data is posted to it.
Here is how you can spot the difference.
If it is just provisioned and you try to run a query against it, the response looks like this:
Inactive Application Insights
However, if there is any sort of data created, then the response looks like this:
Active Application Insights
If this is part of your deployment pipeline, then I came up with a fix (or a hack depending on how you look at it).
By using Application Insight’s REST API, I am posting a custom event via powershell to awaken the Application Insights instance and then I add a wait step by using the Start-Sleep -Seconds 30 command which will wait before it provisions the alert. This will give plenty of time to the app insights instance to wake up and start doing its thing.
Here is the powershell that will add a custom event. You have to provide the Instrumentation Key.
Just wait for 30 seconds after that and you’re good to go.
Find more on my personal blog: http://chapsas.com
