Introducing the Lua Amalgamator for Redis

This week Nubix released an interesting new piece of open source software, designed to make it easier for Enterprise Developers to send large Lua scripts into Redis.

The Lua Amalgamator for Redis, which can be found at github.com/BixData/lua-amalg-redis, is a fork of the original Lua Amalgamator for the Puc-Rio Lua VM for operating systems.

Background

In the realm of web browsers, JavaScript is referred to as the assembly language of the web. New initiatives such as WebAssembly strive to make it easier to virtualize, distribute, and PolyFill for different VMs, while cloud services such as Cloudflare Workers provide the “Lambda” execution environment.

When I first learned of JavaScript being reffered to this way back in 2014, I was a cloud architect at Zimbra, studying Docker, Redis, Heroku’s 12-factor cloud-native bible, and the amazing tooling coming from HashiCorp and Jeff Lindsay. Needing a Redis-based worker library at the time, I happened across qless, and found myself confronting a software library that didn’t really exist in the normal sense, but instead existed only as tiny polyglot language bindings to a large Lua script (github.com/seomoz/qless-core) that is sent into Redis. Having worked extensively with stored procedures long ago, something clicked in my brain, and I started referring to Lua as “the assembly language of backing services”, while cloud services such as Redis (2011) and CloudFlare WAF (2013) started to emerge as execution environments. I started getting the sense that the author of Redis, Salvatore Sanfilippo, had just taken a major step towards realizing Hadoop’s original dream of sending code into the data — except that Redis made it look easy(!!), and it supported over-the-air deployment (which if I’m not mistaken essentially qualifies Redis as a Lambda service). Needless to say, I’ve been a huge fan ever since.

Getting Started

I’ll start by installing amalg-redis on my Mac with:

Next, I’ll write a script that needs some support from the LuaRocks ecosystem. I use Moses, the lodash/underscore Lua equivalent, for just about everything, and so that’ll make a great example.

For testing purposes, you’ll notice that I both print the result, and also return it. Printing the result is for when I’m testing with my local operating system and its Puc-Rio Lua VM. While returning a result is for when I’m sending the script into redis, so that I can see the result.

Now that the script is written and tested locally, I’m going to perform the first of 2 steps of the amalgamation process. The first step is to run the script while amalg-redis traces its execution and accumulates a list of module dependencies into an amalg.cache file.

I’ll now perform the actual amalgamation. The amalg.cache file serves as configuration, while my local LuaRocks repository serves as the source of module source codes that will be merged into a single 2,913 line script.

I’m now able to send the amalgamated script into Redis:

Taking it Further

How far can you take this? I haven’t found the limit yet! I recently trained an Apache Spark K-means model within Redis using Stuart, the Lua-based Spark runtime for embedding and edge-computing.

Final Words

Redis is elegant software that is fun to use and seems to have unlimited uses. It’s my hope that the Lua Amalgamator for Redis opens up some new and interesting use cases for the open-source community, while helping to nudge Nubix one step closer towards its IoT Virtualization goals.