SQS…The feature where it all started.

As for the first story of this publication, it’s always a good omen to follow the chronology of events as it happened, so let’s discuss the very first feature rolled out by AWS back in late 2004, the era of Rackspace and the cumbersome procedures for setting up servers even for one single website.

In words of the official AWS documentation -

“Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message-oriented middleware, and empowers developers to focus on differentiating work.”

So…why was AWS SQS launched in the first place? SQS intended to provide a highly scalable hosted message queue that resolves issues arising from the common producer-consumer problem or connectivity between producer and consumer.

Ok, so what benefits did it provide, you ask. Well, there are plenty.

Reliably deliver messages

Use Amazon SQS to transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be available. SQS lets you decouple application components so that they run and fail independently, increasing the overall fault tolerance of the system. Multiple copies of every message are stored redundantly across multiple availability zones so that they are available whenever needed.

Eliminate administrative overhead

AWS manages all ongoing operations and underlying infrastructure needed to provide a highly available and scalable message queuing service. With SQS, there is no upfront cost, no need to acquire, install, and configure messaging software, and no time-consuming build-out and maintenance of supporting infrastructure. SQS queues are dynamically created and scale automatically so you can build and grow applications quickly and efficiently.

Keep sensitive data secure

You can use Amazon SQS to exchange sensitive data between applications using server-side encryption (SSE) to encrypt each message body. Amazon SQS SSE integration with AWS Key Management Service (KMS) allows you to centrally manage the keys that protect SQS messages along with keys that protect your other AWS resources. AWS KMS logs every use of your encryption keys to AWS CloudTrail to help meet your regulatory and compliance needs.

Scale elastically and cost-effectively

Amazon SQS leverages the AWS cloud to dynamically scale based on demand. SQS scales elastically with your application so you don’t have to worry about capacity planning and pre-provisioning. There is no limit to the number of messages per queue, and standard queues provide nearly unlimited throughput. Costs are based on usage which provides significant cost saving versus the “always-on” model of self-managed messaging middleware.

Ok…so what is in there for a business?

• You are in control of who can send messages to and receive messages from an Amazon SQS queue.
• SSE lets you transmit sensitive data by protecting the contents of messages in queues using keys managed in AWS Key Management Service (AWS KMS).
• To ensure the safety of your messages, Amazon SQS stores them on multiple servers. Standard queues support “at-least-once message delivery”, and FIFO queues support “exactly-once message processing”.
• Amazon SQS can process each buffered request independently, scaling transparently to handle any load increases or spikes without any provisioning instructions.
• Amazon SQS locks your messages during processing so that multiple producers can send and multiple consumers can receive messages at the same time.
• Your queues don’t have to be exactly alike — for example, you can set a default delay on a queue. You can store the contents of messages larger than 256 KB using AWS S3 buckets or Amazon DynamoDB, with Amazon SQS holding a pointer to the Amazon S3 object, or you can split a large message into smaller messages.

Message deletion

SQS does not automatically delete messages once they are sent. When a message is delivered, a receipt handle is generated for that delivery and sent to the recipient. These receipts are not sent with the message but in addition to it. SQS requires the recipient to provide the receipt in order to delete a message. This feature is new as of 2008 where only the message ID was required for message deletion. Because the system is distributed, a message may be sent more than once. In this case, the most recent receipt handle is needed to delete the message.

Once a message is delivered, it has a visibility timeout to prevent other components from consuming it. The “clock” for the visibility timeout starts once a message is sent, the default time being 30 seconds. If the queue is not told to delete the message during this time, the message becomes visible again and will be present.

Each queue also consists of a retention parameter defaulting to 4 days. Any message residing in the queue for longer will be purged automatically. The retention can be modified from 1 minute up to 14 days by the user. If the retention is changed while messages are already in the queue, any message that has been in the queue for longer than the new retention will be purged.

What’s this fuss about FIFO queues?

Let’s summarise the difference in points.

Message Order

• Standard queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they are sent. Occasionally (because of the highly-distributed architecture that allows high throughput), more than one copy of a message might be delivered out of order
• FIFO queues offer first-in-first-out delivery and exactly-once processing: the order in which messages are sent and received is strictly preserved

Delivery

• Standard queues guarantee that a message is delivered at least once and duplicates can be introduced into the queue
• FIFO queues ensure a message is delivered exactly once and remains available until consumer processes and deletes it; duplicates are not introduced into the queue

Transactions Per Second (TPS)

• Standard queues allow a nearly unlimited number of transactions per second
• FIFO queues are limited to 300 transactions per second per API action

Regions

• Standard queues are available in all the regions
• FIFO queues are currently available in limited regions only

SQS Buffered Asynchronous Client

• FIFO queues aren’t currently compatible with the SQS Buffered Asynchronous Client, where messages are buffered at the client-side and send as a single request to the SQS queue to reduce cost.

AWS Services Supported

• Standard Queues are supported by all AWS services
• FIFO Queues are currently not supported by all AWS services like CloudWatch Events, S3 Event Notifications, SNS Topic Subscriptions, Auto Scaling Lifecycle Hooks, AWS IoT Rule Actions, AWS Lambda Dead Letter Queues

Some use-cases for the two can be -

• Standard queues can be used in any scenarios, as long as the application can process messages that arrive more than once and out of order
• Decouple live user requests from intensive background work: Let users upload media while resizing or encoding it.
• Allocate tasks to multiple worker nodes: Process a high number of credit card validation requests.
• Batch messages for future processing: Schedule multiple entries to be added to a database.
• FIFO queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can’t be tolerated
• Ensure that user-entered commands are executed in the right order.
• Display the correct product price by sending price modifications in the right order.
• Prevent a student from enrolling in a course before registering for an account.

Zombie Queues? Dead Letter Queues…

A dead-letter queue is a queue that one or more source queues can use for messages that are not consumed successfully.

The dead-letter queue of a FIFO queue must also be a FIFO queue. Similarly, the dead-letter queue of a standard queue must also be a standard queue. Dead-letter queues are useful for debugging your application or messaging system because they let you isolate problematic messages to determine why their processing doesn’t succeed.

Amazon SQS does not create the dead-letter queue automatically. You must first create the queue before using it as a dead-letter queue.

How SQS work back-end?

Distributed queues

There are three main parts in a distributed messaging system: the components of your distributed system, your queue (distributed on Amazon SQS servers), and the messages in the queue.

In the following scenario, your system has several producers (components that send messages to the queue) and consumers (components that receive messages from the queue). The queue (which holds messages A through E) redundantly stores the messages across multiple Amazon SQS servers.

Message lifecycle

The following scenario describes the lifecycle of an Amazon SQS message in a queue, from creation to deletion.

1. A producer (component 1) sends message A to a queue, and the message is distributed across the Amazon SQS servers redundantly.

2. When a consumer (component 2) is ready to process messages, it consumes messages from the queue, and message A is returned. While message A is being processed, it remains in the queue and isn’t returned to subsequent receive requests for the duration of the visibility timeout.

3. The consumer (component 2) deletes message A from the queue to prevent the message from being received and processed again when the visibility timeout expires.

“Amazon SQS automatically deletes messages that have been in a queue for more than a maximum message retention period. The default message retention period is 4 days. However, you can set the message retention period to a value from 60 seconds to 1,209,600 seconds (14 days) using the SetQueueAttributes action.”

Let’s do a Poll… No? Okay.

Amazon SQS provides short polling and long polling to receive messages from a queue. By default, queues use short polling.

With short polling, the ReceiveMessage request queries only a subset of the servers (based on a weighted random distribution) to find messages that are available to include in the response. Amazon SQS sends the response right away, even if the query found no messages.

With long polling, the ReceiveMessage request queries all of the servers for messages. Amazon SQS sends a response after it collects at least one available message, up to the maximum number of messages specified in the request. Amazon SQS sends an empty response only if the polling wait time expires.

On a concluding note, a service that is extensively used by companies like Netflix, Dropbox, NextDoor and Amazon itself, SQS still stands out as the pioneer king of messaging queues among all its competitors ( IBM WebSphere MQ and Microsoft Message Queuing)even till date since its inception.