This is part 11 of my Node / Redis series. The previous part was Node, Redis, node_redis and NodeRedis.
The beauty of using Node as a full stack developer is that you don’t have to switch linguistic contexts between the front-end and back-end development. I think that is part of the reason why I’ve resisted Lua so much — it introduces another context that I have to think about and manage. With few exceptions, my internal motto has been ‘if you can’t do it with Javascript, it isn’t worth doing.’
A few months back, I had this twitter exchange with Salvatore, in which my dreams of a full, end-to-end Javascript development flow were dashed.
So, after facing the music, life must go on and I still need to make web applications. With a minimal amount of Lua polluting my codebase, and a few tricks, I saved a lot of time.
For this example, let’s assume you’ve got an Express-based app, and you’re using something like Passport or Passwordless that places the username in the req.user. You want to give the user the ability to make changes to their profile information stored in a Redis hash. It looks something like this, where bob is the username.
If you want to create an AJAX-y way to modify each field you might have a route that specifies the field. Then you can post the new value to the field. Your route will look something like this:
app.post('/edit-profile/:field',function(req,res,next) { ... });In your function, you’ll create the key from the req.user, pull the field from the req.params.field variable the get the new value from req.body.
Of course, user input should always be considered dangerous, so you don’t want users just adding arbitrary fields to that hash — so you’ll need a whitelist of fields that are allowable. This is all fine / good if you have a simple case of every user having the same set of fields — but what happens when you have a highly varied set of fields depending on some logic?
In the context of this example, let’s say that paying users get the ability to put a link on their profile and moderators get to make up a novelty title (Grand Poobah of Moderation, etc.). Then, later, as part of a promotion, you give three individual users another field to promote a charity through a profile field. So, now you need to integrate these different user attributes (paying, moderator, charity) and vary the field whitelist against those different attributes.
I have problems with this for a few reasons:
- Callback Hell is starting to show,
- The whitelist is hard coded,
- ‘charity’ only affects a small slice of users but requires changes to the codebase,
- You’re having to keep track of another key (the SET my-authenticated-app:user-abilities:bob),
- Relationships between the ability and the fields are not one-to-one (‘website’ is enabled by ‘paying’, ‘title’ is enabled by ‘moderator’). It’s confusing.
- It’s not atomic.
So, let’s use a little Lua to make this more concise and just better.
The opposite of HSETNX
HSETNX is one of the Redis commands I’ve never had to use. Essentially, it’s a no-overwrite version of HSET — if a field exists on a key, then it doesn’t overwrite it. Simple enough concept, but I’ve never had use for it.
Recently, ZADD got a few more features, namely the NX | XX option. The NX options works somewhat like the HSETNX command.
> ZADD myzset 5 “five”
(integer) 1
> ZADD myzset NX 5 “six”
(integer) 1
> ZRANGE myzset 4 -1 WITHSCORES
1) “five”
2) “5”
3) “six”
4) “5”
> ZADD myzset XX 5 "seven"
(integer) 0From the docs for ZADD:
XX: Only update elements that already exist. Never add elements.
NX: Don’t update already existing elements. Always add new elements.
You may be thinking “This is all informative, but what does it have to do with our AJAX profile editor example?”
Let’s approach the profile from a different way. Instead of making a hard-coded whitelist, let’s make the profile itself act as the whitelist. So, conceptually, any user that has a field present in their profile hash is allowed to edit it. We enable a user’s access to a particular field by adding a blank value to their profile hash. So, if we wanted user bob to be able to specify a title, then we could do something like this:
hset my-authenticated-app:users:bob title “\n”You have a lot of flexibility to make changes this way — you don’t need to change your AJAX field update code to enable a new field to be written. Of course, I wouldn’t administer a website from redis-cli but you could easily write an arbitrary system in your administration pages that allowed adding blank fields to one or many user account/hashes.
How do we make this happen? It would be nice if Redis supplied the opposite of HSETNX (a.k.a. HSETXX, using the terminology from ZADD) — this hypothetical command would allow for an out-of-box implementation. We could do this non-atomically by first doing HEXISTS followed by a HSET if the field is existent. Or, we can use a tiny bit of Lua pollution, make it atomic and keep the javascript clean.
A tiny bit of Lua.
I’ll admit that I’m a novice Lua user. I’ve got just enough knowledge to get in above my head. But anyone with some general programming skill and Redis experience should be able to follow what is going on here.
The biggest leap here is KEYS vs ARGV. I think the article Lua: A Guide for Redis Users nicely explains the basics of Lua Redis scripting in general, but especially does a nice job of making that distinction.
I tend to think of Redis commands accepting arguments, much like a function in Javascript. It is, at least from the Lua perspective, a bit different. You have two categories of information passed to Redis. Keys — which are references to the actual Redis keyspace and Arguments (e.g. ARGV), which can be non-key related information (in data, field names, etc). When invoking those commands you must make a distinction.
As an example, to invoke our newly created command, we can do this.
redis-cli EVAL “$(cat hsetxx.lua)” 1 my-authenticated-app:users:bob car FiatHere we are passing the entire script (stored in hsetxx.lua) into the Lua interpreter. The second argument specifies how many of the arguments to pass as keys (and will end up in the KEYS table) and the remaining will end up as non-key arguments (ending up in the ARGV table).
Here is how this all looks when you bring it together into a simple Express app:
A few notes from the script:
- The rk() shorthand is from one of my other articles — it just joins keys together by colons.
- In the POST route, I’m using regular expressions in the route string itself. This is an often overlooked feature — you can restrict a param by a regular expression — it is a big security win. I’m assuming that all of my fields will be only upper- and lower-case English letters. The route will not match anything with any other characters. As an example:
POST /edit-profile/name //Works!
POST /edit-profile/123 //404
POST /edit-profile/im:a:bad:guy //404
POST /edit-profile/imAGoodGuy //Works!- I’m just catching errors in the response to eval. I’m treating non-valid responses the same as valid. I’m of the mindset that caught invalid responses should not let a baddy know they are invalid, but if you wanted to provide better feedback, you could add an argument on to the callback and conditionally send a status code.
- Seriously, validate and sanitize user input. Even if it is something generic (simple escaping) you’re better off. You might also want to limit the size of the accepted input — remember this is Redis — you have finite space. It is possible that someone might pass very large values taking up that finite space.
Using the original keys and fields from above, let’s run some simple examples of this script using CURL.
#this should work
$ curl --data "value=fiat" http://127.0.0.1:3000/edit-profile/car
{"field":"car"}
$ curl http://127.0.0.1:3000/my-profile
{"location":"London","car":"fiat","hobby":"knitting"}#this will not work
$ curl --data "value=bar" http://127.0.0.1:3000/edit-profile/foo
{"field":"foo"}
$ curl http://127.0.0.1:3000/my-profile
{"location":"London","car":"fiat","hobby":"knitting"}#this will also not work
$ curl --data "value=bar" http://127.0.0.1:3000/edit-profile/foo2
Cannot POST /edit-profile/foo2#add the ability for "foo" to be a field
$ redis-cli hset my-authenticated-app:users:bob foo ""
(integer) 1
# "foo" as a field is now enabled
$ curl --data "value=bar" http://127.0.0.1:3000/edit-profile/foo
{"field":"foo"}
$ curl http://127.0.0.1:3000/my-profile
{"location":"London","car":"fiat","hobby":"knitting","foo":"bar"}
We’ve made an atomic, single-key solution that is flexible and concise by using five lines of Lua pollution in your otherwise clean JS project. I’d rather have less code and a little Lua than more lines and messy JS.
