Reactive Scala Wrapper for Amazon SQS

TL;DR: https://github.com/kifi/franz

The backend for Kifi is built as a fully Service Oriented Architecture. In the context of that, we run into two issues:

1. The System needs to be able to deal with the loss of any machine so long as there are others of the same type. In other words, a machine may be shut down rather abruptly at any time. This is not just to deal with failure but also to allow us to do Continuous Deployment in a seamless fashion. Real time inter service calls can be fairly easily made robust by smart request routing (a subject for another post), but we still needed a way of resuming longer running tasks that are in progress when a machine goes down.
2. We frequently move large amounts of data from one service to another. We don’t want one service requesting or sending a lot of data to be able to overwhelm another, i.e. we needed a mechanism that would allow the producer of some data to produce as fast as it want to and the consumer to consume as fast as it wants to.

Since we run on top of AWS, we chose Amazons Simple Queue Service (SQS) to deal with these two issues. SQS is basically a managed persistent message passing system.

The fact that it’s persistent (for up to 2 weeks) pretty much deals with issue number 2 above, i.e. the producer can just put data into the queue as fast and as much as it wants to and it will happily sit there for the consumer to catch up.

SQS also supports long polling for messages, which makes it easy to reactively read from the queue.

Issue number 1 above is solved by the fact that SQS messages are read from the queue and deleted separately. When a message is read from the queue it is not actually removed, but becomes invisible to clients for a certain (configurable) amount of time. The client who read the message can then process it and once done delete the message. If there is an error (e.g. because the service was shut down in the middle of processing) the message will reappear in the queue and another service can process it.

Amazon provides a very nice Java library for dealing with SQS (which already has reactive elements), but since we run on Scala/Play we wanted something that is easier to use from Scala, as reactive as possible and plays nicely with the Play Framework.

We ended up writing a reactive Scala wrapper for the Amazon Java Library that, in addition to lower level queue operation also lets you have typed queues, so long as the type has a Play! style implicit Json serializer.

For example, suppose you have some type Data with an implicit Play Json serializer (a Format[Data]) sending an object of that type to the queue with the native Java Client looks something like this:

Using Franz it looks like this:

where the send call simply returns a future which will resolve successfully iff the message has been successfully sent and contain an exception otherwise.

You can find the actual library (MIT licensed) as well as technical documentation here: https://github.com/kifi/franz. It’s available from Maven central, so if you are using sbt, just add

"com.kifi" % "franz_2.10" % "0.3.2"

to your dependencies and you should be good to go. All classes are in “com.kifi.franz”.

It’s still in early stages of development, but the existing feature set is quite robust. We make heavy use of it in Production. Feedback in the form of issues, pull requests, etc. very welcome.

We wrote this post while working on Kifi — Connecting People with Knowledge. Learn more.

Originally published at eng.kifi.com on April 25, 2014.