Adopting an Event-Driven Architecture: A comparative look at AWS messaging solutions — SNS

Part II of a four-part series aiming to expose some of the several solutions available within AWS. Access Part I on AWS messaging solutions here.

Occasionally in an event-driven architecture the systems/services that need to exchange messages, form a topology where the point-to-point capabilities provided by a standard queue solution starts to show its limitations.

In this second part of our series, I will cover an AWS service that aims to help us when SQS is not enough: Simple Notification Service (SNS).

To whom it may concern

While a queue provides a powerful way to decouple two participants in a system, it does so by establishing a 1:1 relationship between the producer and consumer as seen in Figure 1.

Figure 1. A point-to-point connection between two applications using a queue.

Sometimes your application has more than one system interested in receiving the same message. How do you solve this problem? Figure 2 illustrates one solution.

Figure 2. Multiple queues receiving the same message published.

As you can see this solution has some limitations:

1. The number of published messages is directly proportional to the number of consumers.
2. For every new consumer, there needs to be a change in the logic of the producer.

Traditional messaging solutions, such as RabbitMQ, offer functionality that enables the Publisher/Subscriber (pub/sub) pattern. In this pattern, the producer publishes the message without specifying a destination and the consumer indicates the interest in receiving the messages by subscribing to them.

Figure 3. A producer publishes messages to an endpoint X and the consumer subscribes to X to start receiving the messages.

This solution solves both limitations we saw previously. The producer does not have to know which queue is used by each subscriber and it only sends the message once.

This allows delivering the same message to multiple destinations, also known as fan-out capability for your application. Let’s see how SNS allows us to implement a pub/sub pattern and more!

Simple Notification Service

As the name implies, SNS aims to provide a simple way to enable the publisher/subscriber pattern for your application. Unlike most traditional solutions, what makes this service particularly interesting is the fact that the subscriber can be more than just a queue, as SNS supports many types of integrations, such as invoking Lambdas, calling HTTP endpoints, sending SMS, Email, and even pushing mobile notifications.

Figure 4. Integrations with SNS.

I will cover the most common types of integration in later sections.

Basics

Topic

At the heart of SNS we have the concept of a topic. This can be thought of as a logical channel where the publisher sends messages and to which one or more subscribers can attach themselves to receive the messages.

As you can see, we only need to specify the message contents and the ARN of the topic.

In a similar way, as we saw with SQS, messages sent to a topic are durable. And so, once you’ve got confirmation it has been received by SNS, it is guaranteed that the message has been persisted in multiple servers. Once a message has been delivered to all subscribers it is removed.

It is important to highlight two things. If you add new subscribers to a topic they will receive messages published at the time of their subscription and onwards. If there are no subscribers to a topic then all messages will be lost until the first subscriber is added.

Another similarity with SQS is that there is no strict guarantee in the message ordering when using the default topic type. If the application subscribing to the topic requires this, you essentially have two options: implement this re-ordering at the application level or choose the topic type to be FIFO, which is only available when the subscription is made to a FIFO queue as well.

Message

The second concept is the message itself which, like in the case of the SQS, contains a body of text and optionally user-defined attributes.

Those custom attributes are in the form of a key-value pair: the name of the attribute and the actual value. The attributes are normally used to hold metadata associated with the message itself and can be leveraged to offer additional functionality, such as filtering.

In this example, the key is the name of the attribute (eventType) and the value is a string with value OrderPlaced.

You can have up to 10 attributes with any message published.

Let’s look a bit more closely at the most common types of subscribers and what limitations each one may have.

Queue

This is likely the most common combination you will find, as it matches closely the traditional message broker model.

Here, your application will publish the messages or events to the topic and one or more subscribers will be SQS queues that will receive these events. The applications will poll those queues and react to these events.

Figure 5. Application publishing an event and multiple queues subscribing to the same topic.

API Integration

This integration enables you to automatically send messages to an HTTP or HTTPS API endpoint. Its main benefit is to remove the need to have a subscriber that would serve just as a passthrough, since each consumed message will be forwarded to the subscriber via an API call.

Figure 6. API endpoint subscribing to a topic.

A common use-case would be when you need to integrate third-party or legacy applications that already expose endpoints to collect the message contents.

When using HTTPS you can even rely on basic authentication by specifying a username/password to be used with each call. When receiving a message your API endpoint will receive a POST call with a set of headers and a JSON payload in the body like the one below:

For security purposes, you have to confirm the subscription of the API endpoint before SNS starts sending you the first message. SNS will make a call to the same endpoint but this time the payload will indicate a Type being SubscriptionConfirmation and containing a URL you must contact to confirm you are ready to receive messages.

It is important to highlight that SNS will retry confirming the subscription like the actual notification of messages following the retries policy you specified. The default is to try up to 3 times and wait 20 seconds between each retry. A retry will be attempted, within the aforementioned limits, if the API returns an HTTP code outside the 2xx-4xx range.

The retry policy can be updated at the topic or subscription level and can control items such as the number of retries, the delay between those attempts and what backoff strategy to be used. A detailed description of all options can be found here.

Mobile Push Notification

Mobile computing is everywhere and SNS helps to provide a platform-agnostic way to integrate your application to devices. Imagine your user wants to be notified when a certain event happens, such as when an item of the wishlist arrives, or when an order has been shipped.

Without SNS, you would likely have to write integrations with the notification services, such as:

• Amazon Device Messaging (ADM)
• Apple Push Notification Service (APNs) for both iOS and Mac OS X
• Baidu Cloud Push (Baidu)
• Firebase Cloud Messaging (FCM)
• Microsoft Push Notification Service for Windows Phone (MPNS)
• Windows Push Notification Services (WNS)

With SNS it can be as simple as with Figure 7.

Figure 7. Different mobile push services subscribing to a topic forward notification to devices.

If you want to leverage this integration, check additional details such as different types of notifications that can be sent and the process to register your mobile application with SNS here.

Topology

Your application will likely contain one or more publishers, and for each one or more subscribers. How you organize and route the messages between them defines the topology and has implications on the setup, development, and maintenance. While the actual topology will vary depending on your application, two common models are:

• One topic per message type

In this model, you have as many topics as you have different message types being sent by the publisher.

Figure 8. One topic per message type topology approach.

This approach is very flexible as it allows the subscriber to simply choose which event types it is interested in creating the subscriptions to the topic. The downside is that you have to manage this big(ger) number of topics and if a subscriber is interested in all events it has to keep adding those subscriptions continuously.

• One topic per publisher context

In this model, you have a single topic used by the publisher to send all messages it produces.

Figure 9. One topic per publisher topology approach.

This topology offers simplicity in management — via a single topic — but implies that all subscribers are interested in all types of messages sent by the publisher. When this is not the case, the simplest approach to handle involves discarding all the messages the consumer doesn’t care about.

Although simple, it is costly, as it will still have to be retrieved and processed by the consumer just to be thrown away.

For that reason, a better solution would be to remove the messages you do not want before they are delivered to subscribers — be it a queue or one of the other possible targets of an SNS topic.

Enter the filtering capabilities of SNS!

Filtering

The filtering capabilities of SNS aims at providing you with options that push some of the cost/complexity out of your application and into the infrastructure.

A filter is defined by the subscriber of a topic by specifying a policy that if met will make the message be used. The qualifier is defined in terms of the attribute name(s) and possible value(s) of interest.

Boolean logic is used when evaluating the policy. If you have a single attribute specified then only messages that contain at least that attribute will be considered.

If your policy refers to more than one attribute, then only messages that contain all the attributes will be considered.

In the previous example, the same message would not be delivered to the subscription because it does not match all the attributes of the policy.

Within a given attribute you can either specify a single value or a list. If you use a list then any of the values will be used for the match.

For example, imagine I have an Order Management System (OMS) that is responsible for the order lifecycle of a company. As part of its functionality, it publishes messages of the changes associated with an order: once it is placed, once it gets paid, etc. On the other side we may have, among the several applications that care about orders, a Warehouse Management System (WMS) that only cares when an order is in very specific states.

Let’s define an attribute called eventType and as values put OrderPlaced, OrderApproved, and OrderCancelled accordingly.

In this case, the Inventory could subscribe to the Orders topic but specify it only cares about the approved or canceled orders. The following policy illustrates this.

This is by far the most common case, but SNS also supports more complex filters, including matching an attribute value against a prefix, a range for numeric values, and even a blocklist that accepts messages with all but a certain value. For more information, you can check this page.

Handling Issues

Failures are part of any application and that applies to SNS as well. As a developer, you should always look for what are potential problems and how the service will handle them.

For SNS this largely involves issues when trying to deliver the message to a subscriber. In this case, a retry policy in place varies based on the type of service integrated. For example, retries to SQS subscriptions are attempted more than 100,000 times over the course of 23 days. A full list can be found here, so make sure you understand what will happen in your case.

Additionally, to understand how the retry will happen, it is recommended that you set up a dead-letter queue for your subscriptions. This way if the retry policy says SNS should no longer attempt to deliver a given message, it will be forwarded there, where you can monitor and decide what to do about it.

What’s next

By now we have covered two solutions that can be used separately or together to support a wider range of patterns to connect the various systems/services that your application needs. SNS extends the traditional message broker’s capabilities by offering integration with APIs, SMS, and even mobile devices.

In part three of this AWS messaging solutions series, I will present EventBridge and how it differs from the solutions seen so far!

Editorial reviews by Liela Touré & Pablo Martinez

Want to work with us? Click here to see all open positions at SSENSE!