High Performance Erlang Cassandra Client — Marina
Cassandra is a battle tested widely popular NoSQL database. It is proven to utilize commodity hardware and designed to handler large amounts of data. At redBus we have sufficiently large Cassandra cluster which is storing huge amounts of data. Dig at “Data Flows” for some of the technical details around it if you like. There is a custom thin layer in front of Cassandra, which is written in Java to write and read from the cluster in addition to small amount of processing and data transformation. We have noticed that under certain load this layer starts failing and is unable to keep up with large number of concurrent connections.
An activity was initiated to quickly prototype and see whether a similar layer written in Erlang is able to keep up with massive amount of concurrent requests. Although, this was not particularly hard but the initial choice of CQErl (most complete Erlang client) proved to be less efficient. It was then Marina came into the picture. Although, there erlclass, but it depends on the DataStax C/C++ driver for Apache Cassandra making it the second choice in case Marina failed to impress us.
Marina is an amazingly fast and light weight Erlang CQL client for Cassandra. Although, there are a couple of other older alternatives but Marina stands out among the rest by providing a very clean and light weight interface to talk CQL. There was one issue with Marina though — It has no support for authentication. This was quickly solved with help of a quick hack, wherein the approach taken for org.apache.cassandra.auth.PasswordAuthenticator is simply used. This simplistic approach allowed us to quickly work with a custom API layer written in Erlang/OTP which proved to be massively scalable and tolerant to load. This along with Marina made the prototype a huge success.
Final Notes
CQErl is a great and feature rich Erlang client for CQL, but it falls short on performance by a huge margin. Another alternative; erlclass appears to be highly performance (based on some benchmarks), but since Marina worked its magic, so we started using it instead. Marina is worth checking out to anyone looking for a high performance, straightforward and light weight Erlang client. It is very easy to setup with very simple and straightforward configuration settings.
