Visualize Azure Activity logs on Elastic-Kibana stack !

Abhimanyu Garg
Jun 11, 2018 · 4 min read

As Cloud adoption is growing, it has become more important than ever to monitor your cloud resources, activities in your cloud accounts and track the events happening within your services.

Azure tracks all the events in your Azure Account/Subscription and publishes it to Azure Activity Log service. As the number of events grow it becomes really difficult to filter these logs and translate them into readable information.

Image for post
Image for post

This is where ElasticSearch-Kibana stack makes life easy. All you need to do is stream these logs to Elastic Search service and then use Kibana to visualize the logs.

Architecture (Local ELK Stack — Elastic-Logstash-Kibana)

Image for post
Image for post

The above architecture shows ELK stack setup on a Linux or Windows VM in a public subnet.

Setting up ElasticSearch, Kibana and Logstash is not in scope of this article. We will be covering the setup in a separate article and linking here once it is ready. Lets get started-

Steps :

  • Login to the instance where Logstash service is running and add the following to the input section of the logstash config file-
input {
http {
codec => "json"
port => "8080"
}
}

The following configuration should be present in output section to push the ingested data to elastic search.

output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "azure-%{+YYYY.MM}"
document_type => "%{[@metadata][type]}"
}
stdout {}
}

This configuration enables http-input-plugin to listen to requests coming on port 8080 and output the data to Elastic Search endpoint. Make sure this port is open on the VM firewall inbound ports as well as Network Security Group.

  • Start the Logstash service. (service should keep running in background)
Image for post
Image for post
  • Once Namespace is created, Click on the +EventHub option to create EventHub.
Image for post
Image for post
  • Open App Services and click on +Add to create Function App.
Image for post
Image for post
Image for post
Image for post
  • Select the newly created App Function and click on + to create a Function. Choose Custom Function from right pane-
Image for post
Image for post
  • Select Event Hub trigger from the set of triggers. Enter values as shown below. Click on New for selecting Event Hub Connection-
Image for post
Image for post
  • Select Connection values as below and click on Create.
Image for post
Image for post
  • Now open the function and add the below code to the run.csx file and save it.
using System;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System.Net;

public static void Run(string myEventHubMessage, TraceWriter log)
{
String data = myEventHubMessage;
log.Info($"C# Event Hub trigger function processed a message: {data}");
data = data.Replace("http://schemas.microsoft.com", "");
data = data.Replace("http://schemas.xmlsoap.org", "");
JToken token = JObject.Parse(data);
//var parsedData = format_json(data);
string url = "http://<Logstash Server Public IP>:8080/";
foreach (var record in token.SelectToken("records"))
{
Console.WriteLine("record:--------------------------------------------------------------------");
Console.WriteLine(record.ToString());
string result = ",";
using (var client = new WebClient())
{
client.Headers[HttpRequestHeader.ContentType] = "application/json";
result = client.UploadString(url, "POST", record.ToString());
}
}
}
  • The above C# code uses Newtonsoft.Json library which is not included in Function App by default. So we need to include those libraries in our Function. Add a file by name Project.json and paste the below code and save it.
{
"frameworks": {
"net46":{
"dependencies": {
"Newtonsoft.Json":"6.0.0.0"
}
}
}
}
Image for post
Image for post
  • Function configuration is now complete. Now open the Azure Activity log service and click on Export.
Image for post
Image for post
  • The configurations are complete. Logs will start flowing from Activity logs Service –> Event Hub –> Function App –> Logstash –> Elastic Search
Image for post
Image for post
Image for post
Image for post

For more details and other articles on Azure -

TechManyu Azure

TechManyu

TechManyu is a great platform to share your technical…

Abhimanyu Garg

Written by

Cloud and DevOps Professional with keen interest in System Design & Architecture. SRE | MultiCloud | Kubernetes (CKA) | Microservices | Automation | Ops

TechManyu

TechManyu

TechManyu is a great platform to share your technical knowledge and expertise to help millions of other developers and technical guys out there. Building an interactive community is our ultimate goal through which everyone can share and benefit at the same time. www.techmanyu.com

Abhimanyu Garg

Written by

Cloud and DevOps Professional with keen interest in System Design & Architecture. SRE | MultiCloud | Kubernetes (CKA) | Microservices | Automation | Ops

TechManyu

TechManyu

TechManyu is a great platform to share your technical knowledge and expertise to help millions of other developers and technical guys out there. Building an interactive community is our ultimate goal through which everyone can share and benefit at the same time. www.techmanyu.com

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store