System Design — Choosing Between AWS Kinesis and AWS SQS

In this blog post, I explore AWS Kinesis vs AWS SQS, showing the benefits and drawbacks of each system and which one to prefer based on your system design requirements. It highlights the difference between data streaming and queueing.

Why AWS Kinesis

AWS Kinesis Streams enables large scale data ingestion and real-time processing of streaming data. It provides ordering of records, as well as the ability to read and/or replay records in the same order.

• Allows record size up to 1 MB
• Works at shard level not message level
• No auto scaling, developer needs to track shard usage and re-shard the Kinesis stream when necessary
• Limited read throughput (5 transactions per second per shard)
• Number of shards in a stream determines the maximum throughput
• Multiple consumers can be attached to a single stream, and each consumer can process every record individually (thanks to shard-iterators)

Why AWS SQS

AWS Simple Queue Service (SQS) offers a reliable, highly scalable serverless hosted queue for storing messages and easily moving data between distributed application components.

• Allows relatively smaller message size up to 256 KB
• Each message can be processed independently
• Auto scales to dynamically increase read throughput by scaling the number of tasks reading from the queue
• Provides messaging semantics (such as message-level acknowledgment/fail) and message visibility timeout
• 120,000 limit for the number of inflight messages for a standard queue and 20,000 for a FIFO queue

AWS Kinesis v/s AWS SQS

Message retention

• Kinesis — Stores records for 24 hours by default, can retain streaming data for up to 7 days
• SQS — Can configure message retention period from 1 minute to 14 days, default is 4 days

Message retry

• Kinesis — No delivery guarantee, service consuming data can retry as long as data is present, consumer doesn’t remove data, ordering is also guaranteed in case of re-drive items
• SQS — At-least-once delivery guarantee, continues to republish unacknowledged message until the message reaches max age as per Message Retention Period

Failure handling

• Kinesis — No Dead Letter Queue (DLQ) out of the box, every application that consumes the stream has to deal with failure on its own
• SQS — If consumer fails to process message within visibility timeout, the message is sent to a configured DLQ and can be received again in SQS

Number of consumers

• Kinesis — Supports multiple consumers capabilities, same data records can be processed at the same time by multiple consumers or different time by the same consumer
• SQS — Supports only single consumer at a time, once a message has been acknowledged it’s deleted from the queue

Ordering of records

• Kinesis — Supports in-order processing within a shard however, no kind of ordering between shards can be guaranteed
• SQS — Standard SQS queue doesn’t guarantee in-order processing however FIFO queues support it but with limitations

Routing of records

• Kinesis — Related records can be routed to the same record processor
• SQS — No support for routing

Other AWS services support

• Kinesis — Can send stream records directly to services such as Amazon S3, Amazon Redshift, Amazon ElasticSearch, Splunk, AWS Lambda
• SQS — Other services can be integrated through AWS Lambda

Sample use-cases

• Kinesis — Log and Event Data Collection, Real-time Analytics, Mobile Data Capture, Internet of Things Data Feed
• SQS — Application integration, Decouple microservices, Decouple live user requests from intensive background work, Batch messages for future processing

Hope you liked this post and it helps you decide which Queue-based AWS service to use for your system design!

Here are some next steps you can take:

1. Checkout AWS Documentation for their recommendation on when to use Kinesis vs SQS!
2. Send some claps my way!
3. Follow me on Medium & Twitter for more tech blogs!