Dodge the busy work: Using lodash and async to write tighter Node.js / Redis apps

This is part 17 of my Node / Redis series. The previous part was Reds Internals: Searching and better searching with Node.js and Redis.

I hate repeating myself. I think that most programmers share this sentiment — repeating yourself creates error-prone code, busywork and bloat. There are many ways around it but I think if you work with a particular API or syntax for long enough, you get more sophisticated in eliminating repetition. Something seeps into your brain and it triggers this itch to remove any vestige of repetition. Recently, I found myself writing lots of little wrapper functions for the Redis API in Node (using node_redis) and I thought there must be a better way. Using a dash of functional programming and a smidgen of lodash and a pinch of async you can tighten up your code and make it perhaps more readable.

Let’s take a common task in Redis — a two-step lookup. In my case I have a bunch of IDs that act as aliases to hashes located at different keys. So, first you have GET the value then feed that value as a key into an HGETALL. Not too complex of a process.

My IDs are stored as simple strings, something like this:

> SET myapp:by-id:1000 myapp:pets:my-hash
> SET myapp:by-id:1001 myapp:pets:my-second-hash

The hashes look like this:

> HMSET myapp:pets:my-hash name "Button" species "Canis lupus"
> HMSET myapp:pets:my-second-hash name "Flipper" species "Tursiops aduncus"

Writing this in Node.js, the code you’ll probably come up will likely resemble this:

This is perfectly workable code, but I wouldn’t say it is ideal. The first thing you’ll likely notice is that this code is pretty specific, which is fine for a small app, but if you’re building something more complex you’ll be writing slightly different versions of the same functions over and over, so maybe you want to generalize. Also we’re getting into callback hell territory. You might also notice that we’re repeating ourselves when defining the rKeys.

Let’s address rKeys. We’re using rk to prevent quoting busy work — but we are also repeating the namespace (myapp) for each key. Seems fine for now with two keys, but when you get a few dozen, you’ll be annoyed by it and the chance of a typo in a string literal increases. Since rk takes a variable number of arguments, we can use the lodash function _.partial to tighten this up. _.partial applies some of the arguments to a function and returns a new function. Here is how we can use it for our keys:

The two console.logs will output:

myapp:by-id:1000
myapp:pets:button

If you think this is more complicated than the original — you’d be right. Later, we’ll see there are some distinct advantages to having your keys as functions.

Using async, we can cut down on the callback hell (sometimes I think the async module is too literally named, perhaps it should’ve been called “remove-callback-hell”). What we are doing in the original code is a two-step waterfall — the return of the first function is being fed into the arguments of the next. The async.waterfall function can do this with a more attractive / readable syntax, as well as reusable way. It also takes care of worrying about any err results in the chain, passing them along to the complete callback if any are non-null.

Looks a bit better, but we can keep going. Now, we’re going to try to do some syntaxical optimization on the two functions inside the waterfall. Node_redis uses prototypal inheritance internally and the this keyword is used extensively. We’re going to need to manage that for the next step — to do this we can use lodash’s _.bind function. This will make sure that any further monkeying we do in the code will not move the cheese on this.

Not much change, but this will allow us to manipulate the commands without having to bother with the this context. Now, lets bake-in some arguments, similar to how we were using _.partial for the rKeys object. We’ll use _.curry as opposed to _.partial, as _.curry allows us to specify an arity. This means after a certain number of partially applied arguments are hit, we’ll get a function returned. In this case, we’ll use curried function to apply the first argument alone inside the waterfall array. This will return back a new function that only has a callback to the array used in the waterfall. We didn’t really need the outer function for the bound HGETALL function, so we can ditch that too.

Now, let’s make the getKeyFromAlias a little more high-level. Remember when I told you that having functions that return your keys would be useful? Here is how we can use it. Lodash has a function called _.flow, it is a bit like async.waterfall, but just not asynchronous — you feed the return of one function into the arguments of another and it returns a new function (if you ask me, it should have been named _.functionalCentipede). Any arguments you give to this newly created function will be fed into the first function specified in the _.flow arguments.

Now, this is likely where I would stop in the ‘real world.’ It’s built of reusable parts and readable without going too crazy. Anyway, you’re likely to have to do something else to an input or an output. So, stop while you are ahead.

However if you’ve had a few cocktails and feel like coding, you might write something that goes even further. It dispenses with the function definition entirely, ditches the _.curry, _.flow, and async.waterfall. Instead, we’ll use async.seq, which operates as a waterfall but returns a function — anything you pass into it will asynchronously redirect the return callback values into the arguments of the next — so it’s like if async.waterfall and _.flow had a baby. The other trick we will use is to wrap our rKeys.alias function with async.asyncify as which creates a dummy callback-style function around a non-asynchronous function.

Bringing it all together we end up with:

Comparing the initial version to the final version, you’ll notice that the code is actually longer, but you do gain a bit. You’re also loading more libraries (lodash, async), so the advantage can seem a bit muddy with this code. Keep in mind this is a pretty small example, once you start layering on app logic, you’ll likely thank yourself down the line for adopting a more generalized, reusable approach — being able to reuse your bound Redis calls and keys-as-functions it will enable you to plug together these building blocks to hide complexity that might otherwise manifest as a tangle of nested functions.