Thinking in events — working with aws lambda and serverless architecture
Event sourcing is not a new concept. In the ear of the serverless architecture world, events trigger nearly everything in lambda.
If you had worked with worker pattern or actor pattern, you would find lambda are just another actor/worker but acting at the service level.
The Event Sources:
You can find a long list of event sources for lambda here. (https://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html)
They generally come down to 3 major categories:
1. Notifications (e.g. SNS, S3 etc.)
2. Queues (SQS is a newly available option)
3. Event Streams (e.g. Kinesis, Dynamodb stream etc.)
But When Should I Use What?
It’s a question I often get from discussions. So Here are a few high-level rules I use for myself. Most ideas are not limited to just working with lambda either.
1.SNS based lambda
1.1 What is SNS?
SNS is a fully managed Pub/Sub messaging service of aws. Just like any sub/sub mechanism, there are topics and subscribers. A over simplified thinking, we can regard events such as S3, API Gateway, etc., to be the AWS builtin SNS.
They are almost real-time, which will trigger your function instantly in an async fashion.
It is also better to assume it as fire and forgets. Once notification got published, as far as SNS concerns, its job is DONE.
Subscribers are responsible to handle the notification and fulfil intended logic.
SNS does handle retries in different ways when it comes to different subscribers. (https://aws.amazon.com/sns/faqs/) However, relying on SNS retry are NOT safe.
1.2 a few considerations when using SNS in lambda:
1. how many events are you publishing concurrently?
2. how important is your event delivery?
The first question is to do Lambda’s concurrent execution limit, which is 1000 by default. (see more: https://docs.aws.amazon.com/lambda/latest/dg/limits.html) So if you have topics, which have very bursty nature and high throughput. Using SNS + Lambda may not be so ideal, as your execution could be throttled and lost.
The second question is how timely do you want your change to be reflected. In some cases, we have a lot of redundant events, which triggers the same logic. If you have a lot of redundant events it may not be a bad thing. Redundant events could be your friend, as retry would naturally happen to ensure the desired function is triggered.
2. SQS based Lambda
2.1 SQS stands for Simple Queue Service. It is a newly released event source to lambda.
The benefit is evident, as now you can dictate what to do when your handler was failing in processing message. You can decide whether a message is worth retrying, and how to handle those retires.
The nature of being a queue also means every message from the SQS event source can only be successfully consumed once by a single listener before dequeuing
2.2 SNS and SQS Combo FTW
If reliable message delivery is critical, and, you also need pub-sub mechanism, SNS+SQS is a winning combo!
SNS guarantees the delivery to SQS. With queue in place, you can handle all your retries through the queue mechanism.
Say you have a new image saved to s3, you want to preprocess the same image into different sizes. One for thumbnail; one for retina; one for low-res. With such set up you can speed up your image processing in a parallel way.
The other benefits of this set up are reducing the concurrency when SNS is bursty. less risk of throttling and message loss.
3. What are the benefits of the event stream?
A stream is a different type of event source. I will take Kinesis for example for the following. Capacity wise, comparing to queue, stream usually can handle a much larger throughput more efficiently.
You can push records to a stream very similar to the queue. Records will be saved in a timed order.
The major difference between stream and queue lies in the consuming and processing records. Once a record is processed, instead of deleting the record, consumer moves its checkpoint timestamp forward.
As a result, Stream can handle more than one consumers at the same time concurrently. (e.g. Kinesis can take up to 5 concurrent consumers.)
We can also replay events from a given time point or reprocess the whole stream if needed. Such feature is especially useful in disaster recovery or debugging (rerun) issues.
The stream is powerful. When using the stream as the event source for lambda. We need to watch out for stream characters. Oftenly, It will come down to how you handling faults in your solution. When the error happens in record processing, lambda will keep retrying, until times out. This means checkpoint would not be able to move forward along the timeline, causing delay or blocking.
The other aspect of using stream, is, consumer processes records in batches. If one record within the batch errored out could cause the whole batch of records being reprocessed. How do you handle redundant events, or do you need events to be processed exactly once are key considerations before you start adopting stream mechanism.
You can use all of them as well
There is an infinite combination in terms of utilizing different types of event sources. Serverless is turning a new page for event-sourcing and microservices. Here is a POC example, how to use, SNS, SQS, and Dynamo Stream, creating a task scheduler, 100% serverless. https://github.com/zdjohn/sns-boomerang
Happy coding!~
