Visualize your Azure Sentinel data with Grafana

Each day we collect more and more data. Making sure we collect -all- the relevant data is by itself already a daunting task, but the effort is meaningless if we can’t make it actionable. Machine learning, like available in Azure Sentinel FUSION, makes things actionable in an easy way.

However, to support your Security Operations Center (SOC), it helps to visualize and provide a clear picture on the data as it flows in. While Azure Sentinel has out-of-the-box dashboarding capabilities, it also works great with third party solutions.

In this article I’ll show you Grafana, and the Log Analytics connector that Microsoft provides for Grafana, to visualize your Azure Sentinel data.

Dashboarding

Dashboarding has been around for a long while in Microsoft’s security solutions. Already ten years ago, in 2009, I presented about Microsoft’s Audit Collection Services (ACS), part of System Center Operations Manager (SCOM), at Teched North America, and how it could help visualize the security status of your environment using dashboards.

Now that organizations are moving from SCOM to Azure Monitor, they can start using the more advanced dashboarding capabilities that Microsoft Azure provides. With the just released AIOps public preview Microsoft provides you with so called Dynamic Thresholds.

“Dynamic Thresholds you no longer need to manually identify and set thresholds for alerts. The alert rule leverages advanced machine learning (ML) capabilities to learn metrics’ historical behavior, while identifying patterns and anomalies that indicate possible service issues.”

One thing to understand is that security dashboarding is different from dashboarding in availability and performance management. Here’s what my quote from a recent presentation I did:

“Security dashboards are different than IT operations dashboards: security searches for outliers (because those are risky) while in Ops if all goes well you are green (99%). — Maarten Goet”

Therefore, Azure Sentinel provides specific security dashboards out-of-the-box. Some of them are solution focused (Office 365), some are technically focused (Insecure Protocols) and some are geared towards third parties (F5, Palo Alto, etcetera). Technically, these are JSON files that work in the Azure Dashboards section of your portal.

Microsoft regularly updates it (GitHub) repository with new versions of the dashboards as they receive feedback from the field. You can manually update the JSON file in your Tenant or use the built-in functions in the Azure Sentinel UI.

Grafana

Another popular choice to visualize data from Azure Sentinel is to use open source visualization tools. Grafana calls itself the open platform for beautiful analytics and monitoring. Grafana is a great option, because it has a large ‘store’ with visualization types (most of them free), and because Microsoft provides you with a native Log Analytics connector for Grafana.

With that connector, you can use Kusto (KQL) queries to get specific data from Azure Sentinel and map it onto one of Grafana’s visualizations. For instance, a world map with network connections, or a list of Alerts. Grafana has dashboarding features that most SOC’s will love, for instance the rotating dashboards.

Grafana is a very versatile platform and goes beyond just traditional dashboarding. It has features for sending notifications, annotating dashboards, mix & match data sources, and much more.

What about Kibana?

Kibana is another popular open source tool that helps you visualize and understand trends within log data. Kibana is the ‘K’ in the ELK stack, the world’s most popular open source log analysis platform, and provides users with a tool for exploring, visualizing, and building dashboards on top of the log data stored in Elasticsearch clusters.

Asaf Yigal summarizes it as follows:

“Kibana’s core feature is data querying and analysis. Using various methods, users can search the data indexed in Elasticsearch for specific events or strings within their data for root cause analysis and diagnostics. Based on these queries, users can use Kibana’s visualization features which allow users to visualize data in a variety of different ways, using charts, tables, geographical maps and other types of visualizations.

The key difference between the two visualization tools stems from their purpose. Grafana is designed for analyzing and visualizing metrics such as system CPU, memory, disk and I/O utilization. Grafana does not allow full-text data querying. Kibana, on the other hand, runs on top of Elasticsearch and is used primarily for analyzing log messages.

If you are building a monitoring system, both can do the job well, though there are still some differences that will be outlined below. If it’s logs you’re after, for any of the use cases that logs support — troubleshooting, forensics, development, security, Kibana is your option.”

PRO TIP: OverOps has done a great side-by-side comparison of Kibana and Grafana.

Easily available from the Azure Marketplace

There are a couple of ways to get Grafana. Grafana has their own hosting option called GrafanaCloud which is free for 1 user and 5 dashboards. On their download page you can also find the binaries for Windows, Linux, Mac, and there is even an ARM option. And yes, there is a Docker container.

However, Microsoft’s Azure Marketplace also offers a Grafana image. Just pick a VM size and it will install Ubuntu (latest) together with Grafana (latest). The default port on which Grafana is published is port 3000.

Azure integration out-of-the-box

Microsoft has built a Grafana data source for Azure Monitor, Azure Log Analytics and Application Insights. It supports both getting metrics directly from Azure Monitor and/or Azure Application Insights for creating ITOps availability and performance management dashboards.

However, the data source also supports connections to Azure Log Analytics workspaces and fetching results from KQL queries. And because Azure Sentinel is based on a Log Analytics workspace, the data source can work out of the box with security data from Azure Sentinel.

PRO TIP: Writing queries in Grafana is made simple with the familiar IntelliSense auto-complete options you’ve already seen in the Azure Log Analytics query editor.

The value is in the Plugins

One of the strong points of Grafana is the fact that they support plugins. Some are built by Grafana, but there is a strong community out there as well.

Some plugins provide you with a ‘panel’, a new way of visualizing the data, while others are ‘data sources’ allowing you to connect to a new data repository. Grafana also has plugins they call “Apps” that are essentially a bundle of panels, data sources, dashboards and new UI pages.

For instance, Grafana built an App for Kubernetes. No need to figure out how to connect, what data to retrieve, and how to visualize: the App takes care of all of that. Just point it to your Kubernetes cluster and you’re good to go.

Let’s visualize Azure Sentinel data

The scenario we’ll be visualizing is about potential malicious connections. Azure Sentinel not only stores IP addresses etcetera, but also the longitude and latitude for the connections which we’ll be using for our dashboard.

Here’s the Kusto (KQL) query that shows the data from Azure Sentinel:

After you’ve set up Grafana, you need to add the Azure Monitor data source and configure a connection to your (Log Analytics workspace based) Azure Sentinel environment:

After this, add a community World Map plugin from here. Create a new dashboard, add the World Map panel, and open up the configuration of that panel. Select ‘Azure Log Analytics’ as the service, select your Workspace (the name of your Azure Sentinel environment) and past in the KQL query:

The result, a world map showing potential malicious connections and their geo location:

PRO TIP: My fellow MVP, Stefan Roth from Switzerland, wrote a step-by-step blog on Grafana in combination with PowerBI and Azure Monitor.

You can find the Grafana dashboarding on my GitHub repository.

What about my other data?

As discussed earlier in this article, Grafana can also be used to visualize ITOps data such as availability and performance management. Marc van Eijk, Senior Program Manager on the Azure Stack engineer team, also used Grafana to build an Azure Stack Uptime Monitor.

Want to geo visualize the people pounding on your SSH port in real-time? No problem. Want to see your DNS analytics? There is a Grafana plugin for that.

PRO TIP: Use a Raspberry Pi to build a display for all of your data graphing needs. A step-by-step tutorial can be found here.

What key indicators will you be reporting on?

Simon Persin makes a great point:

“The challenge for any organization when defining key risk indicators (KRIs) for cyber security is that it is different for every enterprise. There is no blueprint to use as guidance; no one KRI that is pervasive or generic across all businesses, or even industry sectors, because the variances of what needs to be considered are diverse.”

Every business needs to understand for itself which type of attacks or risks could affect it most significantly. Only once those risks are identified can a potential detection strategy be put in place to highlight whether the risk is starting to occur. Even then, the way in which an attack could occur will also depend on the structure and setup of the company itself.

One helpful resource in this regard is MITRE’s ATT&CK framework.

Happy dashboarding!

— Maarten Goet, MVP & RD