Introduction to Message Queuing

What is a Message Queue?

A message queue is a form of asynchronous service-to-service communication used in server-less and micro-services architectures. In message queues, the sender and receiver of messages don’t need to interact at the same time, messages are held in queue until the recipient retrieves them.

A Message Queue

Benefits of Message Queues

Message queues can be used to decouple heavyweight processing, to buffer or batch work, and to smooth spiky workloads.

Better performance

Message queues enable asynchronous communication, which means that the endpoints that are producing and consuming messages interact with the queue, not each other. Producers can add requests to the queue without waiting for them to be processed. Consumers process messages only when they are available. No component in the system is ever stalled waiting for another, optimizing data flow.

Increased Reliability

Queues make your data persistent, and reduce the errors that happen when different parts of your system go offline. By separating different components with message queues, you create more fault tolerance. If one part of the system is ever unreachable, the other can still continue to interact with the queue. The queue itself can also be mirrored for even more availability.

Granular Scalability

Message queues make it possible to scale precisely where you need to. When workloads peak, multiple instances of your application can all add requests to the queue without risk of collision. As your queues get longer with these incoming requests, you can distribute the workload across a fleet of consumers. Producers, consumers and the queue itself can all grow and shrink on demand.

Simplified Decoupling

Message queues remove dependencies between components and significantly simplify the coding of decoupled applications. Software components aren’t weighed down with communications code and can instead be designed to perform a discrete business function.

Message queues are an elegantly simple way to decouple distributed systems, whether you’re using monolithic applications, micro-services or server-less architectures.

Types of message queues

Point-to-Point

Point to Point means message(s) is sent from one application(producer/ sender) to another application(consumer/receiver) via a queue. There can be more than one consumer listening on a queue but only one of them will be ale to get the message. Hence, it is Point to Point or One to One.

Publish/Subscribe

On the other hand Publish/Subscribe is a messaging model where a message is sent to multiple consumers(or subscribers) through a topic. The topic is the link between publisher and subscriber. The subscribers may or may not acknowledge the published message.

Point-to-point vs Pub/Sub

Message Queue Servers

Message queue servers are available in various languages. For example Erlang (RabbitMQ), C (beanstalkd), Ruby (Starling or Sparrow), Scala (Kestrel, Kafka) and Java (ActiveMQ). Selecting a particular Message Queue Server depends completely on the use-case.

Starling

• Written by Blaine Cook at Twitter
• Starling is a Message Queue Server based on MemCached
• Written in Ruby
• Stores jobs in memory (message queue)

Kestrel

• Written by Robey Pointer
• Starling clone written in Scala (a port of Starling from Ruby to Scala)
• Queues are stored in memory, but logged on disk

RabbitMQ

• RabbitMQ is a Message Queue Server in Erlang
• Stores jobs in memory (message queue)

Apache ActiveMQ

• ActiveMQ is an open source message broker in Java

Beanstalkd

• Written by Philotic, Inc. to improve the response time of a Facebook application
• In-memory workqueue service mostly written in C

Amazon SQS

• Amazon Simple Queue Service

Kafka

• Written at LinkedIn in Scala
• Used by LinkedIn to offload processing of all page and other views
• Defaults to using persistence, uses OS disk cache for hot data (has higher throughput then any of the above having persistence enabled)
• Supports both on-line as off-line processing

ZMQ

• The socket library that acts as a concurrency framework
• Faster than TCP, for clustered products and supercomputing
• Carries messages across inproc, IPC, TCP, and multicast
• Connect N-to-N via fanout, pubsub, pipeline, request-reply
• Asynch I/O for scalable multicore message-passing apps

EagleMQ

• EagleMQ is an open source, high-performance and lightweight queue manager.
• Written in C
• Stores all data in memory and support persistence.
• It has its own protocol. Supports work with queues, routes and channels.

IronMQ

• IronMQ
• Written in Go
• Fully managed queue service
• Available both as cloud version and on-premise