Well, I was hoping you’d tell me you’ve tried it and it works ;) It seems like this would need a redis cluster config (https://redis.io/topics/cluster-tutorial), however that comes with the caveat that it required host networking mode, which is not yet supported in docker-compose files v3.
We’re using redis with celery so MongoDB is not an option, unfortunately.
One thing I am going to try and see is if redis allows for an immediate ack to the node a work item is pulled from, if you can afford to do that. (We could.) You’d lose some guarantees but if that’s ok that might allow for simple scaling.