Work queue with Go and RabbitMQ

We need a messaging system

To accomplish what we want to do, we need a system that can work as a queue of messages. We want to have a distributed work queue system, where the workers and the work producers would live on different servers. There can be many source of work and many workers processing those. To build such a distributed system, we need a centralized message queue / message broker. We need a system where we can pass messages and this system will deliver these messages to our workers.

Understanding RabbitMQ concepts

There are several concepts we should keep in mind while working with RabbitMQ.

A command line example

In this post, we really don’t want to go into building a full fledged web app with file uploads. We want to keep it short. So we will do things on the command line. We will build a command line publisher tool which will publish/produce messages. And a consumer which will consume messages. We will then run multiple instances of the consumers in parallel to show how we can scale this system by adding more workers.

Initial setup

Before we can start building out million dollar calculator, we need to have RabbitMQ installed somewhere we can connect to. For us developers, what’s a better place than localhost? Let’s install RabbitMQ on our local machine and have it running.

go get

Building a consumer

We created a consumer directory inside which, we will build our consumer app. The consumer app needs to connect to RabbitMQ, declare a queue it wants to listen to and then start consuming messages.

➜  consumer git:(master) ✗ ./consumer
2019/02/23 20:54:55 Consumer ready, PID: 36361

Building a Publisher / Producer

Now let’s create a producer that will generate random numbers and send to the add queue.

➜  publisher git:(master) ✗ ./publisher
2019/02/23 21:09:59 AddTask: 221+345
➜  consumer git:(master) ✗ ./consumer
2019/02/23 20:54:55 Consumer ready, PID: 36361
2019/02/23 21:09:59 Received a message: {"Number1":221,"Number2":345}
2019/02/23 21:09:59 Result of 221 + 345 is : 566
2019/02/23 21:09:59 Acknowledged message

Empty Exchange Name and Queues

We publish messages to exchanges, consume from queues. In our publisher example, we didn’t specify a exchange name. If the exchange name is empty string, RabbitMQ directly delivers the message to the queue passed as the queue name.

More Advanced Usages

In our use case, we have used a very simple named queue. RabbitMQ exchanges can do a lot more. There are several types of message exchanges which can help us fanout messages (deliver same message across multiple queues) or do topic based matching while delivering messages to queues. Exchanges and queues can do intelligent, configurable message routing to serve complex use cases and build advanced distributed systems.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store