Azure Queue Storage: The Basics
While looking for a cost efficient queue that enabled me to push a high volume of messages, consume them when I wanted to and only pay for the storage of these messages, I came across “Queue Storage”.
Queue Storage is an Azure Service (https://azure.microsoft.com/en-us/services/storage/queues/) that enables you to put messages on the queue and asynchronously process these messages.
Therefore Queue Storage is an ideal method to process events that don’t require a specific order. Some examples:
- Users upload images, and you want to create thumbnails from them
- A horizontal scalable crawler
Pricing
You only pay for the storage used, and the operations you perform on this queue. On the time of writing, the most expensive price was 0.0633EUR/Gb for RA-GRS (Read-access geo-redundant storage) storage , 0.0169EUR/Gb geo-replication data transfer and 0.0068EUR/10.000 operations cost
This makes that if we have a queue of 10Gb that polls messages every second through 3 worker processes that our cost for one day would be:
price storage: 10Gb * 0.0633 = 0.633Eur
worker processes: 3
cost per operation: 0.0068
run frequency per worker: second
seconds in a day: 86400
cost per day for operations: 0.058752 * workers = 0.176256Eur
Total: Storage + Operations = 0.809256Eur (see note 1 and 2)
Note: For the operations you actually have 2 classes, with the second one being half of the first one. However to make this article easier, only the most expensive one was used.
Note #2: We assume instantly 10Gb, however messages come in and get deleted as soon as they are consumed. Therefor we should also assume that our storage will be closer to the not consumed messages = 0 if our consumers work as expected
More info about the types of storage replications (LRS, ZRS, GRS, RA-GRS): https://docs.microsoft.com/en-us/azure/storage/common/storage-redundancy
Advantages/Disadvantages
Advantages
- Cheap
- Only pay for storage and operations → no long running cost such as an Event Hub or a Service Bus
Disadvantages
- No ordering of messages → They can be randomly received
- No subscribe system → You need to poll to detect new messages
- Maximum message size of 64KB
- Maximum TTL (Time To Leave) is 7 days (this is our retention time)
A better comparison between a Storage Queue and a Service Bus can be found here: https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted
Component Architecture / Concepts
A Storage Queue works just as you would expect. You use a library that abstracts an API, which will do REST calls to a storage layer where messages are stored on. The only real difference with a Blob Storage system is that a Queue Storage system has a retention time of 7 Days, and a size limit of 64KB
