Azure Event Hub logging, monitoring and alerting

Introduction

Azure Event Hub is an event ingestion service and a big data streaming platform. It is highly scalable and capable of processing millions of events per second. Azure Event Hub is simple, secures the real-time data and can easily connect millions of devices across platforms.

With this quick introduction, let us understand the following key concepts

Event Producers

Event producers can send events via protocols like HTTP or AMQP. The events sent from the event producers are moved to the Event Hub and each Event Hub is partitioned, which ranges from 1 to 32 partitions.

Partition Key

The partition key helps in processing the events in order.

Event Consumer

Event Consumer reads data from an Event hub.

Create a Namespace

The very first step in creating an Event Hub is the creation of the Namespace with the details below

• Subscription- Choose from the available
• Resource group- Use existing or create new
• Name- to identify
• Location- as per user base
• Pricing tier- The Standard tier of Azure Event Hubs provides features beyond what is available in the Basic tier.

The following features are included with Standard

• Longer event retention
• Additional brokered connections, with an overage charge for more than the number included
• More than a single consumer group
• Capture
• Kafka integration(Event Hubs for Kafka)
• Throughput Units — controls the Event Hubs traffic. A single throughput unit allows 1 MB per second of ingress and twice that amount of egress. Standard Event Hubs can be configured with 1–20 throughput units. Auto-inflate enables you to start small with the minimum required throughput units you choose.

Once Event Hub namespace is created, Event Hub can be created with the following configurations

• Name
• Partition count
• Message Retention period

Sending Events

Events can be sent to the Azure Event Hub by using Visual Studio 2019.
Here .NET is used to send messages to Event Hubs,

• Create a Console Application.
• Install NuGet Package, MicrosoftAzureServiceBus to enable connecting to Event Hub

Add the following using statements,

using System.Threading; 
using Microsoft.ServiceBus.Messaging;

Add the following in your code,

private const string connectionString = "<YOUR_CONNECTION_STRING>";
private const string eventHubName = "<EVENT-HUB-NAME>";

You can acquire the connectionString from your newly created Event Hub Namespace. Under your Namespace click on Shared access policies and then click on RootManagerSharedAccessKey. Get the connection string.

You can add the following code under the Main method,

var eventHubClient = EventHubClient.CreateFromConnectionString(connectionString, eventHubName);
while (true)
{
   try
   {
      Console.WriteLine("{0} > Sending message: {1}", DateTime.Now, "My Event");
      eventHubClient.Send(new EventData(Encoding.UTF8.GetBytes("My Event")));
   }
   catch (Exception exception)
   {
       Console.ForegroundColor = ConsoleColor.Red;
       Console.WriteLine("{0} > Exception: {1}", DateTime.Now, exception.Message);
       Console.ResetColor();
   }
   Thread.Sleep(200);
 }

Once you run the console application, the events are sent to the Event Hubs.

Send Events to Event Hub from Serverless360

A smarter alternative to the solution above is the Send Event Automated task for Event Hubs in Serverless360. One can simulate a real-time scenario by triggering a huge number of events to an Event Hub without writing a single line of code. The Serverless360 GUI encapsulates the underlying complexity and hence improves operational efficiency.

How Do I Monitor My Azure Event Hub?

Azure Event Hub can be monitored from Azure portal using the Azure metrics or you can even use Serverless360 to perform monitoring. Now lets have a deep discussion on both of these monitoring offerings.

Monitoring Azure Event Hubs in Azure Portal

Microsoft Azure offers entity level monitoring where the monitoring is done based on the metrics enabled. Incoming requests, outgoing requests, and successful requests are captured under the metrics section. You can also filter those metrics based on the aggregation value. For example, you can filter your Incoming requests based on the count.

Logging in Event Hubs

Azure Event Hubs logging provides information about the operations done under the Event Hub namespace. Azure Event Hubs have two types of logs namely, Activity Logs and Diagnostic Logs.

Activity Log

Activity logs capture all the actions that are performed on a task and these logs are always enabled. You can view the activity log under the Event Hub namespace.

Diagnostic Log

Diagnostic logs capture well-provided information about all the actions and operations that take place under the Event Hub namespace. A diagnostic setting specifies a list of categories to perform logs and metrics to be collected from the resource and one or more destinations to stream them.

To create a Diagnostic log,

• Click on Diagnostic settings under the Monitoring section.
• Click on Add diagnostic setting

• Under the Category details, you can choose the log that is to be enabled
• In the Destination details, you can choose your destination

To know more about event logs and configurations related to it check, Logging events to Event Hubs

Monitor Event Hubs in Serverless360

Serverless360 monitoring for various Azure resources likeLogic Apps, Function Apps, Service Bus Queues, Topics, Event Hubs, Event Grid, etc., Even though Azure provides the resources level monitoring on the metrics, the actual need would be Consolidated monitoring at the application level. To monitorAzure Event Hubs from multiple perspectives, Serverless 360 provides single Monitor Setting to give you the better performance and Status reports, along with the support for various Notification channels.

Performance Monitor

With the right choice of metrics, we can set up alerts when Event Hubs state violates the desired value with notification channel configured in Monitor Setting, it is also possible to monitor the status and size of the partitions, efficiency, performance, and consumptions of it.

Serverless360 permits configuration on the extensive set of metrics to meet the real-time monitoring requirements.

Status report

Status report provides health reports at a specified time in a day representing that state of Azure Event Hubs against the desired values of its state. Serverless360 is the only monitoring solution that provides partition monitoring for the Azure Event Hubs, which is not possible through Azure Portal.

Challenges faced by Azure users in the Azure portal can be addressed by using Serverless360.

How Do I Check My Data Usage on Event Hub?

You can use Azure metric page or other third-party tools like Serverless360 to monitor on the data usage on event hubs. The data usage can be monitored on the namespace level or even at Event hub level.

Conclusion

Azure Event Hubs help in processing millions of event data and monitors the metrics at the entity level. Serverless360 provides consolidated monitoring at the application level. In Azure, only a couple of metrics can be configured to be monitored but in Serverless360 you can monitor entities on multiple metrics. Enhance the performance of your Event Hub with Serverless360

Originally published at www.serverless360.com on May 29, 2020