Micro Service Events (AWS EventBridge vs. SNS)
During the last few years, we’ve been busy rebuilding the Homegate platform in an event driven, micro service architecture based on AWS. One of the most important buildings blocks, or shall we rather say the mortar that connects the building blocks, is the way our services send events to each other.
While Amazon offers its notification service SNS since more than 12 years; in 2019 it started a newcomer in the messaging bus field by introducing EventBridge. This was very interesting to us as we’re using SNS a lot.
As EventBridge was introduced and gained traction we asked ourselves what advantaged we would gain by switching our SNS based service glue to EventBridge.
Maybe this turned out to be the wrong question for us. Let’s look at our system to see why.
Micro Services Connected by SNS
Homegate is connecting dozens of services via SNS right now. Those services range from single purpose Lambda functions written in TypeScript to database heavy services using Aurora or DynamoDB and Step Functions sending events to other services. Some of the services are also interfacing external APIs such as Salesforce or SendGrid. Here our Lambda functions can call APIs when needed. On the other hand, we have web services to be called as APIs by other systems and end users. This is mostly achieved via API Gateway.
A typical service connects to the system by subscribing to events needed to fulfill its purpose. Those inbound SNS events are then ingested via SQS allowing for orderly data processing and DLQs to handle errors. Outgoing results are pushed back to the system in form of events to a SNS topics.
As Marko Savic has pointed out, we’re also utilizing S3 to store and replay those events if needed.
If we would replace the SNS topic by streams on the EventBridge we would initially gain the same functionality by utilizing SQS as well. However, EventBridge has a different focus. Let’s look at the differences in general specs first.
Amazon SNS Specs
(not to be confused with our own Homegate “Search Notification Service” utilizing Amazon SNS.)
- Amazon SNS is a messaging bus based on publish/subscribe pattern
- lots of consumers can subscribe (up to 12.500.000 subscribers)
- is comparably low latency (median latency: 25 ms)
- listeners can define filters based on topics (and message attributes, but not on payload content)
- ordering: unordered — but can be configured as FIFO (throughput will then be reduced)
- security via: AWS KMS (Key Management Service) & server side encryption
- message size: up to 256 kB
- USD 0.50 / 1 M messages
Amazon EventBridge Specs
Is intended to connect micro services based on different AWS Products as well as external web services e.g. via API Gateways or AWS Lambda.
- default event bus already connected to ca. 100 AWS services
- target services define filter rules to connect to events (max. 5 targets per rule | max. 100 rules per region)
- filtering based on pattern matching in entire content (payload in JSON key detail)
- median latency: 560 ms
- ordering: unordered
- payload size: up to 256 kB (ingestion of max. 10 events per message)
- filtering/security via input transformer
- 1 US$ / 1M events
Even though those general specs are quite different in some regards (such as latency and pricing); both services match all our of requirements for most services. But the specs are not really showing the biggest difference between the two.
The Heritage of EventBridge
EventBridge was first introduced as “CloudWatch Events” which was intended to allow monitoring based on log output. This is the reason for its filtering and pattern matching capabilities. It also brought with it a way of registering and discovering different EventBridge schemas. As it evolved the events of most existing AWS services where added to the schema registry.
Each SNS topics can be imagined as single message bus allowing the publication and subscription of messages by each micro service component (such as a Lambda or a Step Functions). Those messages are then concerning one specific, well: “Topic”.
On the other hand EventBridge is using one central default event bus per account. Therefore it’s important to manage and filter the different types of events.
Schema discovery
We think the idea of a central schema repository for all events generated by our system is great. It automatically provides documentation and and gives an overview of the available events.
However, in order to make use of the schema repository the services need to interface it by generating and publishing them. We looked into the existing integration in the AWS console UI and especially the way to generate and push the schemas automatically to the registry.
The effort of generating and publishing the schemas (e.g. via CDK) turned out to be quite high and the UI for managing our schemas (in the AWS console) seemed to be very basic and missing features.
After all we where just planning to keep our own services to communicate with each other. We are also not using EventBridge extensively to connect to AWS services. Therefore the initiative to use it for new services has stalled and it seems unlikely that we will refactor existing services from using SNS to EventBridge for the time being.
Conclusion
At the current state EventBridge seems to be the perfect choice when allowing external services on AWS to connect deeply into a system.
The right question to ask when deciding wether to go with EventBridge seems not to be:
“What can we gain by switching our SNS based service glue to EventBridge?”.
It is rather:
“Do we have (external) services that need to integrate with our events and that will profit greatly by the schema discovery?”.
But for us, we will keep an eye on it (and the schema registry) and keep on using SNS as the glue of our own building blocks for the time being.
