Event Grid Domain, Cosmos Graph Database, Azure Functions — And Scalable event routing for Graph Events

With serverless platform, one of the foremost requirement is to optimize application performance with better solution construct and leverage serverless platform to its fullest potential. In this article we will describe how to use cosmos database change feed events along with azure functions and event grid domain to create highly scalable enterprise solution which can scale its eventing process to more than thousands event subscriber.

To give you introduction to Azure Event Domain, Azure Event Grid domain is Microsoft event routing service and provide single event routing endpoint backed with multiple event domain topic which works on publish-subscribe mechanism .

In each event domain topic we can have multiple event subscriber endpoint which can be either internal or external applications. Event is published to the subscriber endpoint based on the match and filter condition associated with event subscriber application. Event grid domain also provide us the layer of security through Azure Active directory authentication .To learn more about event grid domain, you can look at Microsoft documentation link . https://docs.microsoft.com/en-us/azure/event-grid/how-to-event-domains

In this article we will take an example of social connect website using cosmos graph database, azure functions and event grid domain as middleware for event routing. To listen for cosmos events you can either implement custom interface using Microsoft.Azure.DocumentDB.ChangeFeedProcessor library or use Azure Functions. In this case we have gone with Azure Functions to listen for graph database change feed and translate the feed information into events which then can be published to the event grid domain.

The overall scaled event routing architecture for cosmos database change feed is shown below

Solution Design

In the first step we will create partitioned collection in cosmos graph database. To do this we will try our hand with azure cli bash command and create a collection with base RU/s unit which is 400. This base throughput is good choice for initial development and test on cosmos graph collection and we can always scale up the initial setup later based on our requirement.

Creating Cosmos Graph database instance with Azure CLI

To create cosmos database resource we can execute below bash CLI command. In below azure CLI command socialconnect is the name of resource group and serverless03 is the name of cosmos graph instance. In this azure cli command we have also defined the read/write paired region (Australia Southeast Australia East) for cosmos database instance.

az cosmosdb create — resource-group socialconnect — name serverless03 — capabilities EnableGremlin — locations “Australia Southeast”=0 “Australia East”=1 — default-consistency-level “Session” — enable-multiple-write-locations true

Once create cosmos resource CLI command is executed successfully, this will create empty instance of cosmos database instance in the specified resource group. Next step is to create graph database and collection in cosmos resource instance through azure CLI command.

To create database in cosmos resource we can use below azure cli command. In command socialconnectdb is the name of the database and serverless03 is the name of cosmos database resource.

az cosmosdb database create — name serverless03 — db-name socialconnectdb — resource-group socialconnect

When command get executed successfully, the bash window will show the resource details for database like id of the database, collection path, etag etc. Here we just need to note that the id of the cosmos database is same as the supplied name in the azure cli command

In the final stage we will add partitioned collection to the cosmos graph database. To do this you we will execute below Azure CLI command. In the CLI command we have set physical partitioned key as users which will be used to distribute graph node and edges into multiple logic partition within single physical partition. Here in the example you can also verify the base RU/s unit along with collection name which is socialcollection.

az cosmosdb collection create — collection-name socialcollection — name serverless03 — db-name socialconnectdb — resource-group socialconnect — partition-key-path /users — throughput 400

Once bash CLI command is executed, the output will contain information about the database collection. This information include id of the collection, path of the document in the collection, different policies associated with the collection and so on as shown in below image.

Create Event Grid Domain Resource through Azure CLI

Similar to cosmos database instance we can create event grid domain through Azure portal or using Azure CLI command. Creation of Azure Event domain using azure portal is easy and can be accomplished through regular search on Azure portal blade and then populating the required fields in event grid domain blade as shown below

Here to give better understating we will follow azure CLI approch to create event domain resource. The snippet of the command to create event grid domain is listed below. In the command socialconnecteventdomain is the name of event domain, socialconnect is the name of resource group and resource location is Australia Southeast.

az eventgrid domain create -g socialconnect — name socialconnecteventdomain -l “Australia Southeast

The response to the above command will contain event domain details which include event domain endpoint url ,resourceGroup name , provisioningState ,name ,inputSchema Type etc.

As we have created the cosmos database and event domain resource in our resource group, next step would be to create a gremlin enabled graph API to perform basic CRUD operation like creation of vertices and joining two or more vertices through relationship etc

Create Cosmos Graph Using Web API and Gremlin.Net

In this step we will create .net core web API using gremlin.net driver to perform basic crud operation on graph node .The web api will also include controller method to manage relationship between nodes through adding and deleting edges between connecting nodes.

To do this ,the first step here is to define the data contract for the node (users) and then also create contract for the type of relationship between nodes like Friends, Knows, and Follows etc. Example of sample user class is shown below

For relationship class this will contain source node, destination node details along with type of relationship between two graph nodes. The basic definition for relationship class is like below

To connect with cosmos graph database collection and perform crud operation we need to add reference to gremlin.net driver and then define gremlin client connection property in startup method available in Startup.cs file. The description of overall implementation is shown below for the reference

To leverage dependency injection for graph API using .net core we will add gremlin client as service in configureservice method implementation. Here in this sample web API we have enabled open API definition to work with swagger file.

This will help us to import the definition in logic Apps custom connector and use it as build in connector within Logic Apps workflow. We can also import the open API definition in API Management for any external access.

Once initial setup is done for start-up class we also need to populate cosmos graph connection details like graph database hostname, key, collection name, database name, partition name into applicationsettings.json file. After updating the required cosmos graph connection properties, we can perform database connection test through visual studio debug console.

In the next step is to add required controller and methods for users and relationship. The method will use users and relationship contract to perform POST, GET and DELETE operation on graph Node and edges. The overall method implementation code is listed below

Create User vertices in Cosmos Graph database

Get User vertices from Cosmos Graph database

Delete User vertices from Cosmos Graph database

Once all the method implementation is completed we will run the web API in local environment and look for open API definition .The open api definition should contain two methods for relationship and three methods for users operation on graph database using gremlin.net

After initial round of test on local environment we will then deploy this .net core web API as azure app service. For simplicity we will use same resource group were our cosmos database and event domain resource exits. This will help us to work within single resource group instead of navigating through multiple resource group container and different azure subscription.

Next within the same resource group we will add azure functions and logic Apps to listen for cosmos graph change events.In this example we will add two azure functions the first one will listen for cosmos change feed using default cosmos input binding and use event grid driver to post the change feed document to multiple domain topic. The second azure function will act as event subscriber and sends the event details to notification hub to that registered devices can get notification based on document change feed.

Azure Functions For Cosmos Change Feed and Event Subscription

In this section we will create azure function which will listen to cosmos graph database change feed using cosmos database input binding’s .This Function will map the cosmos graph change feed intoto event domain event and publish the batch of events to event domain endpoint using event grid sdk whenever there is update to cosmos graph collection documents

In this Function each iteration will check the document label and route the event details to different event domain topic based on the label .In this sample we have four different event domain topic which will listen for any vertices change while one topic will listen for the relationship within the graph database.

We have also flattered the response from graph using custom newtonsoft.json deserializar and used JavaScript serialization and deserialization to post the message into event grid domain. The overall code implementation for azure functions using cosmos graph change feed and event mapping is listed below

Azure Functions and Logic Apps as Event Subscriber

In this section we will add event subscriber to the event domain. To leverage serverless platform we will add Logic Apps and Azure Functions as event subscriber for the events published through event grid domain. To hook up function with event grid we will use event grid input bindings which can handle initial validation handshake without wiring any custom code for validation process.

If you require to use webhook validation for functions we can leverage event grid sdk to send the initial validation token as response .The process is already been described in previous article published on medium. The functions code implementation for event grid subscription is listed below

Logic Apps workflow setting for event grid domain is simple, we can wither add webhook trigger or use build in event grid trigger connector to subscribe for events published through azure event grid domain and event topic.

The webhook setup for event grid is described below .The overall process flow is when a new user is created send a welcome email using event details published through azure event domain topic.

The second logic App will use event grid trigger connector and post the event details to event hub .The event information then can be used for any data analysis through stream analytics or related products.

Setup Azure Resources for Event routing Testing

Next step is to deploy azure function App definition in azure and update the cosmos database connection properties along with event domain key and endpoint details in functions application.settings . Internally we have leveraged key vault to secure secret from any external users.

To setup event subscription for functions, you need to update the event grid subscription information in the deployed function definition as shown below

We can now add other endpoint like logic Apps, webhook, event hub or storage account as event subscriber associated with event domain. This process help to highly scale your event process by leveraging in build capabilities of azure. Below image describe four different event domain topic associated within azure event domain.

And each event domain topic will have multiple event subscriber which can be either internal resources or external partner’s endpoints .In our example we have added two subscription to each event domain topic to work with highly scaled event process.

Build cosmos Graph structure with Gremlin API and test Event Routing

In this section we will use our Gremlin web API to create node and build relationship among different nodes. This will also enable us to test our event routing through Event Grid domain and domain topic. The graph structure which we will create through gremlin web API is shown below.

To create the graph in cosmos database we will use open API definition and populate the required parameters through swagger definition .You can also leverage this API as custom connector in Logic Apps workflow or use it within Functions, web App or application of your choice like postman.

We will now publish some data into cosmos graph database through swagger UI .The post request will use user contract and this will create user node in the graph database when post operation is fired

The response will contains user information along with associated logical graph partition information as shown in below image

Similarly we will add other users to the graph database either though Logic Apps workflow or using the swagger UI. Once all users are created we will use vertex id or source and destination node to create relationship between two vertices .Below to this is through swagger UI detail for creating relationship in cosmos graph

The response will contain invertex and outvertex details along with other document properties like in Label out Label and type of relationship. To get you better visibility we have added the response image below

Event Routing Validation

As we already have function configured to listen to cosmos database change feed. The next step is to verify all the event subscriber endpoint and validate the event routing. The first validation we be done on CosmosChange feed function using the function monitoring tab.

Next we will validate event publish and subscribe monitoring using event domain blade in azure portal .It will list the event which are published and successfully routed the event subscriber endpoint.

Next we will validate Logic Apps which are associated with event domain as subscriber endpoint .In below logic Apps we can the overall execution of Logic Apps in real time when event are published to the event grid domain.

The function is associated with event domain as subscriber in this case has posted the event details to external website as shown in below image .Here we now have shown you how you can leverage gremlin.net ,web api ,functions and cosmos graph database to create reactive application in cloud .

Here we complete this article on cosmos change feed with Azure function and Event grid domain ,in next series of articles we will see how you can optmization gremlin graph query performance using different query approach while doing your initial development and architecture design