Sending and Receiving Custom Events with AWS EventBridge Schema Registry

How use Schema Registry to define the format of events used in your custom applications

Source — https://aws.amazon.com/eventbridge/

The scenario

Suppose you are going to build a solution like this in AWS.

The Order service generates OrderConfirmed events. AWS EventBridge routes them to the Email service, which reads the event and sends out an email.

Both services are based on Java. But the Order service runs in a container while the Email service is a Lambda function. The Order service uses AWS Java SDK to publish a custom EventBridge event that has the following format.

The problem with the custom event format

In the future, these services will be built, tested, and operated by two different teams. Hence, the information that crosses each other’s boundaries must be documented and maintained consistently.

For example, the format of the OrderConfirmed event must be consistent across two teams. Both teams should be well aware of the event’s structure, details of each parameter, etc. An upstream change in the event format must not break the Email service.

Having that in mind, how do you make sure that every developer from both teams is:

• Refer to the schema of the OrderConfirmed event from a central place
• Read and understand the event schema in a technology-agnostic way
• Use the event schema to generate code bindings to ramp up the development

Event schema management with EventBridge Schema Registry

The best option is to use EventBridge Schema Registry to discover, create, and manage schemas for the events used by both services.

Having EventBridge Schema Registry in the solution provides the following benefits.

1. Provide a central place to discover, create, and manage schemas for the solution’s events.

Schema Registry allows the service development teams to edit event schemas kept in a central place collaboratively. Once necessary permissions are granted, developers from both teams can discover, create, or edit the event schemas used to build their use cases. That provides everyone a single source of truth for the structure of their events.

2. EventBridge uses OpenAPI and JSONSchema to define event schemas

Eventbridge event schemas are based on OpenAPI or JSONSchema Draft4 specification. Using standard formats like that helps development teams remove the dependency on a particular implementation technology to describe events. Developers can understand the structure of events without going through each service’s implementation technology.

For example, the Order service can be Java, while the Email service can be written using NodeJS. To understand the event format that the Order service produces, the Email service team doesn’t need to go through any Java code. Instead, they can refer to OpenAPI/JSONSchema based event schema definitions in the schema registry.

3. Provides the ability to generate code bindings from event schemas

For all schemas in EventBridge, developers can generate and download code bindings to help quickly build the service implementations. That eliminates the need to write boring but necessary event marshalling and unmarshalling logic on the service side.

The implementation

Now we have a solution architecture in place. Before each team starts coding their services, they need to agree upon the OrderConfirmed event’s schema. Then define that in the EventBridge Schema Registry.

Once the schema is defined, they can generate code bindings for Java using the Schema Registry. These bindings can then be used to send and receive OrderConfirmed events from respective services.

Before we begin

You can find the source code for the completed solution in this GitHub repo.

In this post, I’ll highlight only the steps that matter most at a very high level. You can always refer to the source code for specifics.

Before you begin, make sure that you have already done the following.

1. You have proper access to AWS EventBridge through a valid AWS account. If not, refer to this to get started.
2. Install and configure AWS SDK for Java and AWS SAM Toolkit in your local workstation.

1. Create a schema for the OrderConfirmed event in the EventBridge Schema Registry

To avoid mixing with standard AWS event schemas, I decided to create a separate schema registry called ‘custom-registry’ to house the OrderConfirmed schema.

Once the registry is created, I created a new schema under custom-registry and named it to edu.orders@OrderConfirmed.

Below is the schema definition, which is based on OpenAPI 3.0.

For more information on managing EventBridge schemas, please refer to this.

2. Generate Java code bindings for the Order service

Now we have an event schema for the OrderConfirmed event. Let’s go ahead generate Java code bindings for the Order service. They do the marshaling, and unmarshalling of OrderConfirmed events from JSON objects to JSON-formatted strings and vice-versa.

EventBridge allows you to download code bindings for the schemas defined in the registry. Java, Python, and TypeScript are the languages currently supported. You can do this either using the AWS Console or via any AWS Toolkits provided for IDEs.

I used the AWS Console to generate code bindings as follows.

You can download code bindings with AWS Management Console

3. Write the custom event publisher

When code bindings are in place, let’s write some Java code to create and publish OrderConfirmed events.

The Order service is a Java application that uses the AWS Java SDK to publish events to the EventBridge. Make sure you have installed and configured the SDK in your local workstation as per the prerequisites.

AWS SDK Maven dependencies can be added to the Order service as follows. Note that we are using the SDK v2 here.

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>bom</artifactId>
            <version>2.15.14</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
<dependencies>

The following adds the EventBridge Maven dependency.

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>eventbridge</artifactId>
</dependency>

To add the downloaded code bindings to the Order service, extract the zip file and copy them into the src/main/java folder.

If you open the order-service project using an IDE like IntelliJ, the project structure should look like this.

order-service project structure

Following additional dependencies are required to work with the code bindings in your code. The SDK uses the Jackson library for serializing and deserializing JSON objects.

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.10.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core -->
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-core</artifactId>
    <version>2.10.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-annotations -->
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-annotations</artifactId>
    <version>2.10.0</version>
</dependency>

Add the following method to schema.edu_orders.orderconfirmed.marshaller.Marshaller class. It returns a JSON formatted string of the OrderConfirmed object.

public static <T> String marshal(T value) throws IOException {
  return MAPPER.writeValueAsString(value);
}

When everything is in place, the final code for publishing events will look like this. The code uses the PutEvents method for sending out events to an event bus called ‘custom-event-bus’ defined in the EventBridge.

Refer to the order-service folder in the Git repo for the complete implementation.

3. Generate the consumer Lambda function

The final step is to create the Email service, which receives OrderConfirmed events from the EventBridge. It is an AWS Lambda function written in Java. For this post’s scope, it merely logs out the event summary.

I installed the AWS Toolkit for my IDE IntelliJ to generate a Lambda function that acts as the Email service. The toolkit allows you to choose an event schema from EventBridge Schema Registry when generating a project. To create a new serverless Lambda function, use File > New > Project in the IDE, then select AWS and AWS Serverless Application.

Then you have to specify the application type. Choose AWS SAM EventBridge App from Scratch (for any Event trigger from a Schema Registry)

Select the region and credentials accordingly. Under Event Schema, I found the custom schema I created at the bottom of the dropdown list.

Clicking on Finish generated me a Lambda project. I went ahead and changed it to write the logic for the Email service.

The following is the function I wrote to handle the OrderConfirmed event coming from EventBridge. Notice that the IDE has generated the Java classes for my schema and included them in the handler method. That allows you to have IDE auto-completion and validation.

Source code for the Email service can be found under the email-service directory of the Git repo.

Running the application

To run the solution, follow these steps.

1. Create a new event bus in the EventBridge console and name it as custom-event-bus.
2. Build and deploy the Email service. You can use SAM CLI for this.
3. Open the order-service in IntelliJ and run the class.

If everything goes well to this point, you should see a log entry from the Email service lambda function in CloudWatch logs like this.