Migrating Redis Sorted Sets without losing data
How to keep the data elements intact when moving sorted sets across instances?
I have been working on a task scheduling service which stores some data in Redis Sorted Sets. It works great for usecases where the data entities need to have a “score”, which could indicate priority or something similar.
I had to change the Redis instances (AWS Elasticache) for alterning the instance size and some maintenance activity. This posed a problem since I had to migrate the Sorted Set data from old instances to the new ones that already had sets with the same name, without losing any of the elements in both the old and new instances. Using the normal
MIGRATE command overwrites the exiting key data.
After sifting through the docs and asking around the web, I stumbled upon a command particularly useful in this case —
Computes the union of sorted sets given by the specified keys, and stores the result in the destination set.
ZUNIONSTORE <destination_key> <num_of_keys> <key1> <key2> ... <keyn>
It combines the elements in all the specified keys, adding the scores of the same element by default. This might not always be desirable, but there’s a way to tweak it according to our use-case.
WEIGHTS <weight1> <weight2> ... <weightn> AGGREGATE SUM|MIN|MAX]
WEIGHTSoption, it is possible to specify a multiplication factor for each input sorted set. This means that the score of every element in every input sorted set is multiplied by this factor before being passed to the aggregation function. When
WEIGHTSis not given, the multiplication factors default to
AGGREGATEoption, it is possible to specify how the results of the union are aggregated. This option defaults to
SUM, where the score of an element is summed across the inputs where it exists. When this option is set to either
MAX, the resulting set will contain the minimum or maximum score of an element across the inputs where it exists.
In my case, I was sure there weren’t any duplicates in the two sets. So I could just use the default weights of 1 since there weren’t any collisions.
This command will work on the same Redis instance though, not across instances. To deal with that, I just renamed the sorted set — using
RENAME command — in the old instance, and used the
MIGRATE command to move it to the new instance. Then I could just use
ZUNIONSTORE to merge it in the existing set on the new instance.
How are you using the Redis Sorted Sets? And do you have another way of migrating them?
Please leave a note here or hit me up on Twitter.