Azure Event Domain and Event routing with Functions ,Cosmos DB ,Event Hub and Logic Apps
We have learned from technology evolution .Patterns and practises have seen changes from time and time to evenly distribute data across communicating systems and applications. In the transit phase from Monolithic applications design to granular structure we all have our how to wow moments. The rise of Service oriented Architecture pattern followed with enterprise service bus and Micro-services architecture pattern ,all of these are targeted towards building effective communication channel across targeted systems. With wide range of digital road map one thing which is most important is data .for enterprise data is important more then ever. Storing and transmitting big data set along with its relational properties is critical to build future knowledge base transformation journey for organisations. This requires special effort and with technology advancement it become easy then before to manage these known and unknown requirement.
In this article we will try to give an overview of what has changed in modern technology landscape and why we should care about these changes when developing enterprise grade solutions
- Modern applications relies on processing big data structured and unstructured .People holds information’s and system stores data. Combining knowledge and data effectively is key to success for any enterprise grade solution and business.
- With advancement of cloud computing and development in serverless and FaaS offering, enterprise are willing to leverage the cloud infrastructure for rapid development. This also enables enterprise to scale its frontend applications and backed systems based on customer requirement and demand .Great example of this can be Azure functions or service fabric connected with globally distributed backend service of cosmos database.
- Integration has changed and it is not only targeting enterprise grade system sitting behind the firewall. This is change natural with global distribution of data and wide variety of SaaS products and partners scattered across multiple regions.
- Another change has been in terms of practising agile and better DevOps practise in organisation. Agile has its benefits in terms of people engagement and determining the key objectives to meet customer sentiment and demand .This keep organisations competitive with frequent changes to applications and software to server customer on right time and in better manner.
This list is endless, we will not go in details, and we can find these information on web with single click. Forget to tell that search is another advancement to technology evolution.
Azure Events Grid
Let us bring ourselves back to event and eventing process. Why should we care and why we should learn. Beginning of 2018 Microsoft has announced general availability of azure event grid, an eventing platform which works with native cloud products and services together with custom build applications to publish and subscribe events.
The basic objective of eventing process is to notify the party of interest about change and what has been changed. This is the big step towards building reactive programming interface for applications. With recent announcement on azure event grid, Microsoft has made general availability to bunch of great features to event grid in form of retry policy, dead lettering of events, manual handshake validation with validation uri, support to storage queues and hybrid connection as destination .This list looks great when we want to control events emitted from source framework to interested subscribing applications .Some of the event emitters and subscriber application and systems is shown below.
For devs like us to gain maximum out of eventing platform, azure event grid driver has built in support for multiple language like .NET, GO, Node, Java, Python, Ruby .This means we can choose variety of supported language to work with event platform to transmit and subscribe event from azure event grid .To learn more about event grid ,there are great article available on Microsoft site https://docs.microsoft.com/en-us/azure/event-grid/ and various blogs site on web. Some of the knowledge source are listed below
The community has done a great work and if you are new to event grid and platform, above links will guide you to get started with event grid without going much into descriptive information. In below section we will cover event domain and how you can manage multiple event grid topic with single entry point for your event driven process.
Azure Events Domain
With rise and adoption of azure event grid, Microsoft has also came up with Azure event Domain. Azure Event Domain service is fully managed event broker service and is in preview state. Event Domain multiple integration and data movement use cases. This service enables organization to publish and subscribe events on scale with single publishing endpoint with n number of event domain topic associated with each Azure event domain.
Each event grid topic associated with event domain can again be scaled with multiple event subscriber working with same or different event filter route condition .Azure event domain is great service to use when we need to synchronise and communicate with variety of application within enterprise own domain space or outside customers and partners.
Using the same event grid driver in code we can build our modern enterprise grade event broker middle-ware. For enterprise this help to minimise the overall resource consumption with effective push notification model rather than relying on scheduled pull pattern of communication. If we look closely to the above diagram, it closely resembles of modern enterprise service bus which is again connected with various stateless and statefull services like Service fabric, Azure functions, Logic Apps, storage, event hub etc.
Publish Event to Event Domains
To publish event to event domain we need to create event domain resource in the resource group .This can be done either with azure portal or shell command .The process has been documented on our previous blog post at https://medium.com/@abhishekcskumar/event-grid-domain-cosmos-graph-database-azure-functions-and-scalable-event-routing-for-graph-2421a7009f59 . In this example we will use console application to publish events to event domain and will route the event details to multiple subscriber like cosmos DB, logic Apps, external webhook listeners and event hub.
Receive Event to Event Domains and Route to Cosmos DB
In below code we have used Azure Functions Cosmos database input bindings to publish event details into cosmos collection. The function is invoked through http trigger and perform initial validation handshake with Event grid topic .
Azure Events Domain Topic
Each domain can have one or more then associated topics to route events . Again each topic is been associated with one or more then one subscription with appropriate event filter condition. To show this with example we have created 4 different topics associated with event domain and each topic has single subscription to update the downstream systems .
Once event domain topic and subscription is setup correctly ,In next we will run our console app and fire up events to event domain which is again get routed to different topics based on topic name passed through event payload. To verify azure functions are listing to the events ,we can click on the monitor tab of azure functions and can look into the run details
Other place to look for is cosmos database collection . To verify the event details you can either use storage explorer or use azure portal an run the basic sql select statement against the collection
Other event subscriber are logic Apps ,event Hubs and external web-hook endpoint . To verify event subscription for external endpoint we had registered webhook.site endpoint within event domain topic with manual validation process .As validation is completed earlier we can see the events getting routed to external HTTP endpoint
Like Event Grid Topic ,Azure Event Domain has good monitoring features through portal . We can drill down to each of the subscription and verify the event details. Below is the example of Logic Apps event subscription which shown how much event get published and routed through webhook endpoint of Logic Apps
As event domain service is in preview state ,it is worthwhile to have your hand-on experience on domain topic to and work with custom event schema to publish and subscribe event through event domain .
Code has been published over GitHub at https://github.com/abhi004/eventdomain