Migrating Redis Sorted Sets without losing data

How to keep the data elements intact when moving sorted sets across instances?

Amanpreet Singh
Nov 10, 2017 · 2 min read

I have been working on a task scheduling service which stores some data in Redis Sorted Sets. It works great for usecases where the data entities need to have a “score”, which could indicate priority or something similar.

The Problem

I had to change the Redis instances (AWS Elasticache) for alterning the instance size and some maintenance activity. This posed a problem since I had to migrate the Sorted Set data from old instances to the new ones that already had sets with the same name, without losing any of the elements in both the old and new instances. Using the normal MIGRATE command overwrites the exiting key data.

The Solution

After sifting through the docs and asking around the web, I stumbled upon a command particularly useful in this case — ZUNIONSTORE.

Computes the union of sorted sets given by the specified keys, and stores the result in the destination set.

ZUNIONSTORE <destination_key> <num_of_keys> <key1> <key2> ... <keyn>

It combines the elements in all the specified keys, adding the scores of the same element by default. This might not always be desirable, but there’s a way to tweak it according to our use-case.

WEIGHTS <weight1> <weight2> ... <weightn> AGGREGATE SUM|MIN|MAX]

Using the WEIGHTS option, it is possible to specify a multiplication factor for each input sorted set. This means that the score of every element in every input sorted set is multiplied by this factor before being passed to the aggregation function. When WEIGHTS is not given, the multiplication factors default to 1.

With the AGGREGATE option, it is possible to specify how the results of the union are aggregated. This option defaults to SUM, where the score of an element is summed across the inputs where it exists. When this option is set to either MIN or MAX, the resulting set will contain the minimum or maximum score of an element across the inputs where it exists.

In my case, I was sure there weren’t any duplicates in the two sets. So I could just use the default weights of 1 since there weren’t any collisions.

This command will work on the same Redis instance though, not across instances. To deal with that, I just renamed the sorted set — using RENAME command — in the old instance, and used the MIGRATE command to move it to the new instance. Then I could just use ZUNIONSTORE to merge it in the existing set on the new instance.

How are you using the Redis Sorted Sets? And do you have another way of migrating them?

Please leave a note here or hit me up on Twitter.

Amanpreet Singh

Written by

Inglourious Coder. Perpetual Learner.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade