Create Shareable Application Insights Dashboards with Humio
Application Insights is a wonderful tool for collecting data about your web applications. In just a few steps you automatically get performance data for your requests, including insights into the SQL or HTTP dependencies you are using to quickly detect bottlenecks. And with little effort you can also start tracking custom events that happen in your application to get an overview of how your users are actually interaction with it.
The tools in the azure portal for reading the data are already great and they are getting better all the time. But for day to day use there is a massive issue.
If you are like me, you want to have a dashboard of your application performance and usage on a screen somewhere in your office. Sure, application insights does email alerting, but there is nothing quite like being able to look at a chart and go “Hm, why do my response times spike every day around noon?”. This is the one thing that Application Insights does not do well.
You can create dashboard, but they are not really Application Insights dashboards, they are in fact Azure dashboard with Application Insights information. This is fine, in fact it allows us to combine the Application Insights data with other elements from across Azure. But while such a dashboard can be shared with other users that has access to the Azure portal, there is no way to share the dashboard with anonymous users. That makes it very hard to get it to display on a setup like the Raspberry PI powered TV we are using for dashboards at my office.
Humio
Many of our dashboard are created using Humio. Humio is another fantastic tool for capturing log data of any kind and digging into it as well as creating dashboard. We use for things like capturing data from IoT devices, web server logs and logs from mobile apps.
Humio uses the concept of a Data Space as a “bucket” into which you can throw any log data. To allow you to make meaningful queries of the data, it must be parsed when put in the data space (Ingested is the Humio term for putting data into a data space). Thankfully the most common data types like JSON is already supported.
Humio is extremely flexible and when we hit the brick wall of non-shareable dashboards in Application Insights we starting thinking if we could somehow use Humio to fill that need.
Connecting the dots
Getting data into Humio is quite easy and can be done via a REST endpoint. Getting data out app application insights is actually also quite easy by using Continuous Export.
Application Insights exports data into an Azure Storage Account. The folder structure is based on the type of exported data, and each file placed in the storage account contains lines of JSON objects (Note: The objects are NOT wrapped in an array, they are only newline separated).
This means that we would need to react to each new file in the storage account, read each line and put it into Humio. This sounds like a job for an Azure Function. Below is the code we used for our function.
To get started with this you need a Humio account. If you log into https://cloud.humio.com/ using a google, bitbucket or github account you will have access to your own sandbox data space that you can use for testing.
You will need the place your data space name (“sandbox” if using your sandbox space) and your ingest token into the function code. The ingest token will be displayed when the data space is empty, but can also be found under settings.
You will also need to update the function configuration with the connection string for the storage account containing the exported Application Insights data. Replace only the “<PATH-TO-YOUR-DATA>” part, the rest is needed for the function to known the parameters of the exported file.
The last bit is telling humio how to read the timestamp. Notice in the function code that the HumioData class set the type as “appliction-insights”. This tells Humio to use the “application-insights” parser. But since Humio has no such parser we need to create it first.
In your data space, click the menu and go to “Parsers”
Create a new parser and set it up like the screenshot below:
This tells humio the read the field “context.data.eventTime” as the timestamp of the event and the format of the event. The timestamp is the ISO 8601 that you can select when you click in the “Datetime Format” field, no need to enter the format yourself. Just remember to set the parser type as JSON and the name as “application-insights”.
Create the Dashboard
Now your data should start flowing into Humio. Play around with making queries. The docs can help with the query language, but here is one to get your started:
type=Requests | requestName := request[0].name | requestDuration := request[0].durationMetric.value/10000 | timechart(function=percentile(requestDuration))This will give you a line chart of the request response times divided into percentiles. Notice that the application insights duration values are always in tenths of a microsecond, so we need to divide them by 10000 to get milliseconds. You can read about the data model here: https://docs.microsoft.com/en-us/azure/application-insights/app-insights-export-data-model
When you have a query you want to show on your dashboard select the “Save As” option and save as “Dashboard Widget”. The dialog allows you to create a dashboard or select an existing. Give your widget a name and click “Save”
You now have a dashboard! Let’s share it with the world! Or at least create a link you can use for your dashboard TV. Under the “Dashboards” menu you can locate your dashboard and select “Share”
In the dialog, you can enter a name and click “Create Shareable Link”. In the list you can copy the link and start using it. Remember that since this link allows anyone read access you may want to be careful where you store it.
