HareGo: When RabbitMQ meets Go

Preface

At Blokur, we utilise RabbitMQ for asynchronous communications between micro-services and to manage jobs. AMQP library is the de-facto standard for Go applications, however it is very low level which makes it hard to write flexible and testable code.

In the world of Go, there are a handful of high-level libraries that facilitate communications with RabbitMQ. Unfortunately they didn’t satisfy all of our use-cases, or they were very specific on what they offer which made it very hard for us to work with. So we decided to build our own library.

We needed something that we could quickly start working on for the most important part of our daily jobs: solving problems. Anything around that is a development cost and does not fit into our model.

Enter HareGo

We started thinking about creating our own library with the following characteristics:

1. It should support almost every aspect of AMQP library.
2. Must be well tested and easily testable.
3. Must be easy to use and extendable.
4. Should come with sane defaults that just work.
5. Must be concurrent safe.
6. Good to have the ability to re-queue messages (not only Nack) and be able to manipulate the message before re-queuing.

We are pleased to announce that we have open-sourced this library. At the time of writing this article, we support Go version 1.14 and above.

API

The HareGo’s interface is very simple. You need to create a harego.Client object with the handy harego.NewClient() function. The only requirements of the NewClient() function is the address of the RabbitMQ server. You can configure every aspect of the client, and consequently the underlying AMQP client, by passing harego.ConfigFunc configuration functions provided by the library.

By default, a harego.Client creates one worker, which handles one publish or one consume request at a time. The client gets a random consumer name. The default behaviour sets the autoDelete and autoAck properties of all messages as false. You can however set them up differently if you wish.

You can pass the harego.Workers(3) configuration function to have 3 concurrent workers. This applies on publishing and consuming messages. The library guarantees to deliver the next available message to the next available worker slot.

It’s a good idea to pass a harego.QueueName() config to let the client know which queue it should interact with. The default queue name is “harego”.

The Client is an interface with the following specifications:

Publishing Messages

To publish messages, you can pass a *amqp.Publishing object to the Publish method. The message will be processed by the next available worker.

Consuming Messages

The Consume method is a blocking call that would use a handler every time a new message is received from the broker. It will stop consuming when the context is cancelled or the client is closed.

A handler should return two values: the AckType value and a delay. The delay specifies the duration it should sleep before it responds to the broker, and the AckType value specifies what should happen to the message.

AckType Values

A decision was made to set the “multiple” value of the ack to false in order to respond to only one message in case of multiple workers. In order to make the worker system work, we have to set the prefetch value to at least the amount of workers.

There are four values for this type:

1. AckTypeAck: responds with msg.Ack.
2. AckTypeNack: responds with msg.Nack.
3. AckTypeReject: responds with msg.Reject.
4. AckTypeRequeue: sends the message back to the queue, and responds with msg.Ack. See the Re-queue Messages section for more information.

Re-queue Messages

With the AMQP library you can Nack messages and they will be delivered to the next available consumer. Sometimes this is not what you expect and you need the message to go back to the end of the queue, and be delivered again. With HareGo not only is it possible, but you can also manipulate the message before re-queuing. Here is an example of re-queuing up to 5 times.

Tips and Tricks

Multiple Connections

When using this library, create multiple harego.Client objects to address different aspects of your logic. Say you are reading from one queue, and writing to two queues, create three objects and call the two publishers inside the consumer of the first one:

Testable Code

In order to make it easy to mock the harego.Client, you can create a function type that would return a new client and an error. You can use a mock generating library to create a mock out of harego.Client interface and use it in your tests.

In the following snippet we are using the mockery library for generating the mocks, and testify library for asserting mocks. Of course you can choose the library you are most comfortable with.

Final Words

For more information please visit the library’s repository. If you need any feature request or found a problem with the library, please create an issue within the github portal.

Interested in joining our team? Please head over to this link and apply.