Open Source Clojure Library for Aerospike

Here at AppsFlyer, we’ve been pretty vocal about being a Clojure shop, as well as migrating to Aerospike as our primary database. While being active in open source has been a core principle, building a culture around producing and maintaining great open source projects has been something we’ve been working up to.

That is why we are very excited to announce today an open source Clojure library that we use in production to wrap our Aerospike Java Client — which for us was a great way to demonstrate the expertise we have gained both with working with Clojure and Aerospike through a single service. Just note, that because this is a production and mission-critical service, it is relatively opinionated. That said, we are very excited to share it with the community — and to receive contributions and collaborate on this project. You can read more on how to contribute.

Let’s get funky! Lisp, Clojure and Scala are just some of the AppsFlyer functional programming jam. Join us! >>Learn More

Below is some general info from the README, that you can find in the newly minted project on Github.

The prerequisites for using the library are fairly basic — Java 8 & Clojure 1.8. Among the features provided by the client you could find:

• Conversion of Java client’s callback model into a future (manifold/deferred) based API.
• Exposing the passing of functional transcoders over payloads (both put/get).
• A health-check utility.
• Hooks for automated client-side metrics.

How the client is opinionated:

• Non-blocking only. This means that it only exposes the non-blocking API. You are still able to block and wait for API calls results using derefwhen needed.
• Futures instead of callbacks. Futures (and functional chaining) are more composable and less cluttered. If synchronous behavior is still desired, the calling code can still deref (@) the returned future object. For more sophisticated coordination, a variety of control mechanisms are supplied by manifold/deferred, or via the library using data transcoders or hooks.
• TTLs should be explicit, and developers should think about them. The client forces passing a TTL and not use the cluster (or the Java library’s) default one. Explicit is better than implicit.
• Requires a single client per Aerospike namespace. Namespaces in Aerospike are more than just namespaces, they may have different hardware/performance implications we believe in a single namespace per Aerospike cluster. And so, the client is a per-namespace one.

Known limitations are:

• Currently supports only single bin records.
• Does not expose batch/scan/query operations. Batch reads/writes are supported via get-multiple/set-multiplethough.

Usage Examples:

Create a simple client (single cluster)

And put/get

We actually get back a record with the payload, the generation and the TTL (in an Aerospike style EPOCH format).

Using Transcoders

The library takes advantage of deferred’s ability to compose and allows to configure a :transcoder to conveniently set this logic:

• get Transcoders are functions of the AerospikeRecord instance, not the deferred value of it.
• put Transcoders are functions on the passed payload. They are called before the request is even put on the event-loop.

On get:

On put:

The transcoder here is a function on the payload itself:

The transcoder option saves some boilerplate and can be easily used to do more useful stuff, like (de)serializing or (de)compressing data on the client side.

Advanced asynchronous hooks:

Since aerospike-clj uses a future based model instead of a callback based model, it is convenient to compose complex asynchronous logic using manifold.

By implementing ClientEvents 2 hooks are exposed that are called for each API call: on-success and on-failure.

Those hooks are called with valuable information that can be used, for example, to configure automatic logging or telemetry on your client. Here is an example of such code, that is reporting useful metrics to statsd.
So assuming you have some statsd namespace that can connect and report a statsd server, and some metrics namespace that is used to properly format the metric names:

A few notes on the above code:

1. Passed arguments:
   op-name, op-result and index are strings. They partially used for metrics generation in our case.
   client here is the IAerospikeClient instace. You can use its fields here, or you can even assoc more keys on it when you create it, to be later used here.
   op-start-time is (System/nanoTime)converted here to microseconds and used to measure latency.
2. The code is using the passed arguments to measure latency and format metrics names. You can easily do other stuff like logging etc.
3. Both on-success and on-failure return the results passed in. Although this logic is the last logic that happens to the operations results (e.g. after transcoders are being run), the returned result will be what the calling code gets as a returned value.

Finally, hook it to your client:

user=> (def c (aero/init-simple-aerospike-client ["localhost"] "test" {:client-events (->DBMeter)}))

Resources

Please feel free to get involved and take this project to the next level. Here are some additional resources available for you to get started

• You can also find the client on Clojars
• Check out the quick start guide to get it up and running in your environment.
• You can find the generated docs here, and the advanced docs for implementing asynchronous hooks and implementing your own client.

This library is distributed under the Apache 2.0 License. Find the license here.