Adopting an Event-Driven Architecture: A comparative look at AWS messaging solutions using EventBridge
Part III of the SSENSE-TECH four-part series aiming to expose some of the several solutions available within AWS.
Event-driven architecture is all about decoupling, at least the temporal aspect of your application. In parts I and II of this series I covered two AWS services — Simple Queue Service (SQS) and Simple Notification Service (SNS) — that alone or in combination aim to achieve this decoupling. But, that doesn’t mean that those are the only options we should consider.
As their name suggests, those services aim at providing a simple way of integrating with your applications. This comes at the cost of pushing some functionality, traditionally found in the messaging infrastructure, back to your application.
In this article I will cover EventBridge, which is another messaging solution provided by AWS, that offers some interesting features not available in SQS and SNS.
So, you want to do more?
As your application evolves, it is likely that you will face the need to handle:
- Integrating with third-party services that are not part of your core competencies
For example, you may decide to use software like Auth0 to handle the registration/login of your customers and make sure your application can react to this.
- Having different types of outgoing systems interested in what is happening
For example, you may want to provide streaming capabilities for the data generated by your application.
In both cases, SNS is going to fall short, in that it will demand that you have to add additional components to enable your goal.
Not surprisingly this is where EventBridge can help you!
EventBridge
EventBridge is an AWS service launched in 2019 that offers a serverless event bus functionality. It can be considered a successor to the CloudWatch Events service and allows your application to listen for events emitted by several AWS services. EventBridge uses the same API as CloudWatch Events, and offers additional features, such as integrating with 3rd party SaaS products (Software as a Service), creating different event buses, and using content-based filtering/mapping rules.
The Basics
Event
The message that will be sent, representing a fact that occurred somewhere in your application, on AWS services or supported partner applications, such as ZenDesk, DataDog, and Auth0.
In EventBridge an event consists of a JSON body and metadata. An example event can be seen below:
While a full explanation of each field can be found here, I’ll highlight 3 key fields:
- Detail-type: This can be used to specify, if relevant to your application, the type of content that this event contains.
- Source: identifies which service generated the event. In the case of events generated by an AWS service, it will be named as the service itself. For custom applications, it can be anything you want. A good practice is to specify a fully qualified application name.
- Detail: the actual data of the event in JSON format.
Event Source
The application or service that is responsible for emitting the event. EventBridge supports 3 types of sources:
- AWS services
Several AWS services can act as event sources, emitting standard events as the result of their operation. A full list can be found here. An example of this is connecting AWS SageMaker to EventBridge, which allows your application to be notified when a training job changes state.
2. Custom services/applications
You may have services or applications that emit events and want the other services to be notified and react to said events. This includes the same applications that were using SNS, as described in Part II of this series.
3. Partner applications
Your application may have to leverage third-party services for non-core aspects such as authentication or customer support. AWS has a list of partner integrations that facilitate receiving events from those partners.
Event Bus
Is the pipeline that receives the events from the event source before being evaluated and delivered to the destination.
There are 3 types of event buses:
Default: Always exists and is associated with your account. The events emitted by the AWS services will appear here.
Partner: Created when you enable a partner application to connect with your account. One event bus per partner can be created.
Custom: Your custom applications will use an event bus to emit the events.
Rules
A list of patterns that will be used to select which events from a given event bus should be delivered to a specific destination.
One rule is associated with one event bus and can specify up to five destinations. It is expressed in the form of a pattern that can use elements from the payload or metadata.
You can have up to 300 rules associated with one event bus.
Target
The destination that will receive the event. EventBridge supports several AWS services as targets, including SNS, SQS, API Gateway, Kinesis, and even other EventBridge buses in different accounts or regions. A full list can be found here.
Filtering Events
A reality with many applications is that not all parts of your system may be interested in all events sent. While you could always consume the messages and decide to drop them in your code, having the capability to do such at the infrastructure level helps to make you write less code, and reduces costs and latency as you only have to process relevant messages.
EventBridge offers this in the form of rules, which specify the patterns that need to match to qualify an event to be sent to a target.
The pattern is a JSON document that references the fields you want to use. An example rule can be seen below:
In the example, we are only referring to the fields source, detail-type, and detail. EventBridge will ignore all other fields from the event. If an incoming event contains those 3 fields with values that match the pattern, it will be delivered to the target.
When creating your rules, it is important to highlight that EventBridge handles values in ways that may differ from what you expect. For example, when you have a rule to target a number you have to be aware that EventBridge uses a string representation. This means that values of 100 and 100.0 are different. Another tip is to leverage the arrays to specify the multiple values that can be used to match instead of creating multiple rules.
Imagine in an e-commerce application you want to notify your customers when orders are either shipped or have problems. If your application emits events such as OrderShipped and OrderCouldNotBeShipped you could create the following rule targeting your CRM.
Because it allows you to inspect the entire event structure, you can define a specific match in the event payload. In our example, the rule will only match if the payload contains a delivery element with a value equal to “express”.
Some more complex primitives allow you to create rules that match on a prefix, on a numeric range/limit, and even on negative lists.
Using the same e-commerce scenario imagine you want all events except the ones that indicate that the order has been picked and packed:
For a complete list of patterns see this page.
Sending an Event
Sending events to EventBridge follows a similar trend from the previous services, as it’s something very simple to do.
In the example above we are passing just one event. However, the putEvents API accepts a list in case you want to send multiple events at the same time.
The response tells us the number of failed events, the identifiers for each of the successfully sent events and if FailedEntryCount is bigger than 0, which ones failed with the error.
This way you can know which ones you may have to correct and resend.
Integrating with Step Function
One area where EventBridge excels is in the number of connections, either as source or target for the events. This allows you to create complex applications, while leveraging managed services to do a lot of the heavy lifting for you.
One interesting situation that you may be faced with, is implementing a mix of orchestrated and choreographed sagas. Imagine you decided to leverage Step Functions to do the orchestrated mode.
While you definitely can do that without EventBridge, using the direct integration that exists with Step Function allows you to start the execution of the Saga directly as a result of the event without the need of any intermediary, a Lambda function for example. Also, you can send events to the bus directly from the Step Function.
This reduces the number of integration pieces, and the lines of code you have to write, debug, and maintain, while avoiding the costs of paying for the execution of the step function and the Lambda.
Integrating with SaaS providers
Depending on the nature of your application, you will need to implement features already provided by a third party. Those SaaS providers can offload you with the cost of developing non-business core functionality.
One of the considerations when taking this approach is figuring out how you will integrate those providers back into your application. The two most common ways are via some sort of API/postback or via polling.
Both ways come with the intrinsic concern that you will have to handle the security aspect as you will transit via the internet. Managing credentials and whitelisting fall on you to set up and maintain.
The polling mechanism is usually inefficient. Set up a polling interval too small, and you may have many round trips without new information. Set up with a wider interval and risk receiving data that is too old.
AWS helps by providing a partner program that enables a SaaS provider to publish events from its application to a specific event bus in EventBridge. In this situation, you do not have to worry about the transit of the events from the SaaS to your application as it will happen from within the AWS infrastructure and be pushed to you.
I will never use SNS again… right?
Wrong. While Part IV of this series will discuss aspects surrounding the selection of the messaging service, it is important to highlight that EventBridge is not the SNS killer.
If we do a quick comparison between both we can see that each one excels in certain aspects, so you have to look at your application patterns before ruling one out.
For example, the difference between both services regarding latency indicates that if your application requires a low latency you may have to choose SNS. The same thing applies if you expect a massive fan-out as EventBridge is limited to having up to 5 targets.
On the other hand, if you require filtering on the contents of the payload or want to leverage a direct integration of certain AWS services, as an event source or target, EventBridge will present itself as the better option.
We are almost there…
EventBridge provides an interesting combination of features and ease of use that should be considered for your application’s integration strategy. If you need to integrate a SaaS application that is supported as a partner, it can save you quite a bit of work to set up and secure the connection.
In the next and final article of this series exploring messaging solutions, I will discuss Kinesis and present some heuristics to help you select which of the services should be used for your application. Stay tuned!
Editorial reviews by Deanna Chow, Liela Touré, and Pablo Martinez. Want to work with us? Click here to see all open positions at SSENSE!
