distributed locking for pennies: distrilock

A while ago I needed a distributed locking solution to compensate for some lousy 3rd-party limitations, and I was a bit overwhelmed by the lack of choice. (Feel free to let me discover more in the comments, but please avoid mentioning solutions you have not tried yourself)

Locks are made for sharing

The most common/popular solutions seem to be:

• Apache Zookeper
• etcd, which recently got a v3 released which should be even better
• redlock
• MySQL GET_LOCK() and RELEASE_LOCK(), not a game winner, I do not advise it
• AWS DynamoDB, too expensive compared to the little functionality used

All of the above-mentioned solutions offer a bit more than just distributed locking, or do not even offer proper scalable distributed locking.

The use case for distributed locking seems like a perfect candidate for a microservice, or an application building block.

A simple service providing only distributed locking feature seems neat when one might not want to switch directly to heavy artillery (Zookeper), not at least until variability of the deployment scenario is more clear.

Deja-vu

Where did I see this? A (reasonably*) scalable (distributed) locking implementation?

They say it’s a glitch in the Matrix

But in the Linux kernel of course! I am talking about POSIX (advisory) record locks, that you can find briefly described in https://gavv.github.io/blog/file-locks/#advisory-locking and by using man 2 fcntl(https://linux.die.net/man/2/fcntl).

Additionally, such POSIX locks can be used across different hosts by sharing an NFSv4 mountpoint, achieving a reliable multi-host distributed locking system.

*= NFSv4 does not scale well itself, but whether this is an issue or not depends on the amount of clients and locks of your use-case; for the details of this problem, see https://www.fsl.cs.sunysb.edu/docs/nfs4perf/nfs4perf-microscope.pdf and http://www.pnfs.com/

Giving it a spin in Go

I implemented a service that exposes very basic distributed locking functionality on top of POSIX record locks:

• https://github.com/gdm85/distrilock

It can be used via TCP and websockets (binary or JSON text), see the README and godoc documentation for official documentation.

Epilogue

I have performed some benchmarks (a more extensive post about them might follow up) to determine the performance under various usage scenarios and with different combinations of client and TCP options (timewait re-use, timewait recycle).

No sensible difference has been measured between the TCP implementation and the Websockets/JSON implementation, thus you can choose the latter for a richer set of features in your applications.

each locking session lasts 1s, repeated 5 times

Please let me know about your experience with distrilock and if you use it for something interesting, I would be curious about the performance and eventual bottlenecks you can individuate.