Store Javascript objects in Redis with Node.js the right way

This is part 2 of my Node/Redis series. You can read Part 1, Dancing around strings in Node.js and Redis, Part 3, Using the Redis multi object in Node.js for fun and profit, Part 4 Keeping track of account subscriptions with Redis and Node.js Part 5 Managing modularity and Redis connections in Node.js, Part 6 Redis, Express and Streaming with Node.js and Classic Literature(?), Part 7 Untangling Redis sort results with Node.js and lodash and Part 8 Redis, set, node.

Hey! If you’re reading this, maybe you want to go to RedisConf 2018 for free?

On the face of it, a Redis hash is fairly similar to a Javascript object used for data storage. You have a key, fields and values. Here is some data represented in roughly the same way between Redis and Javascript:

HSET “a-test” “name” “Kyle Davis”
HSET “a-test” “address” “123 Main Street”

and

var
  aTest = {
    name : ‘Kyle Davis’,
    address : ‘123 Main Street’
  };

Pretty much the same, right? Well, it gets more complicated when you increase the complexity of the Javascript object. Take this slightly more complicated object:

var
  aTest = {
    name : { first : ‘Kyle’, family : ‘Davis’},
    address : ‘123 Main Street’
  }

Now, try and represent this with a Redis hash. Go ahead, I’ll wait.

You can’t, at least directly. Sometimes you’ll see people shove another data interchange format into the string value. I really think this is a bad idea.

You might have wondered why I haven’t mentioned the term JSON in this piece yet. Mainly, because I don’t think it has a legitimate place here. It’s pretty common to see libraries try this strategy. Ultimately, it’s has problems.

#bad - don't do this
HSET "a-test" "name" "{\"first\":\"Kyle\",\"family\":\"Davis\"}"
HSET "a-test" "address" "123 Main Street"

The first thing you run into is that Redis doesn’t understand JSON. So, the data you put into your string very well better be something that you’re never going to need to interact with within Redis (perhaps it is possible with a lua script, but I’m not going down that road). Secondly, you’re going to stringify and parse the whole strong to get a single value out of it. This isn’t a problem with something like a name, but if you have a big complicated structure, this could be a big problem. Finally, readabilty — like it or not, Redis needs to be accessible to an admin — if you have a big chunk of un-tabbed JSON you might as well have a binary format. It’s unreadable by someone sitting behind redis-cli.

While I’ve been picking on JSON, the same goes with pretty much every other format (XML or what not).

You could manually transform your data but that involves a lot of knowledge of how the data looks now and how it will look in the future. Also, I don’t have time for that and I’m betting you don’t either

Best of both worlds

So then, aside from banning nested objects, how do you get the best of both worlds?

Enter, flat. Flat is a small npm module that will convert your nested object to a single flat key / value. Back to our previous example, by using flat you can convert your nested objects to something like:

var
  aTest = {
    "name.first" : ‘Kyle’, 
    "name.family" : ‘Davis’,
    "address" : ‘123 Main Street’
  }

With this, you can simply store it in Redis by doing:

HSET “a-test” “name.first” “Kyle”
HSET “a-test” “name.family” “Davis”
HSET “a-test” “address” “123 Main Street”

Now, if you just want to get the family name, you can:

HGET "a-test" "name.family"

You can also do cool things like:

sort myTestList BY *->name.first ALPHA DESC GET *->address GET *->name.first

Try that with JSON!

The Good, Bad, Ugly

Flat is not perfect. I’m not crazy about how it handles arrays. You get into the same situation you have with JSON — Redis doesn’t really understand what is going on and thus won’t be able to retrieve and manipulate it directly. I don’t have the answer for how to deal with that kind of situation, but it isn’t perfect as is.

Flat also has problems with variables that don’t nicely go into a string — so occasionally you’ll get a hash value of ‘undefined’ instead of something being, you know, just not even in the hash. Make sure you fully test your data with flat and make accommodations accordingly.

Finally, as far as performance, Flat may not be the best. JSON.parse and JSON.stringify are built in functions, so they are going to be hard to beat, but you more than make up for it in the flexibility to only get one value instead of the whole chunk of JSON.

Still struggling to store stuff in Redis the right way?

Drop me a line and we can talk about how I might be able to help you get on the right Node / Redis foot.