This is part 13 of my Node / Redis series. The previous part was Simulating a slow command with Node, Redis and Lua.
I’ve been paying close attention to the issue log for node_redis. An interesting issue came up recently, and while it isn’t seemingly a problem with the node_redis module, it is a problem and and one that can be mitigated in Node.
The problem as presented by cekimy on github:
I’ve meet a problem several times. redis.get cost much time, more than 1s (even more than 1 minute)
This is a terrible online disaster, so i want to know how to set timeout of get command.
Or what could cause redis cost so much time?
I inquired a bit more about the problem and cekimy responded:
…Most of the time, it only takes ~1ms, but this >1s situation came about 1time/4–5months, and continue several seconds.
i have 4 redis servers, each key will hash to one of them, and then redis.get(This caused delay).
Our service has about 500,000,000+ requests one day, and each redis server has 10000+ operation per second on the rush hour.
The server is usually stable running, this problem appears suddenly, and disappears a few seconds later.
Terrible online disaster indeed, especially at that scale. I can’t speak to what could cause the Redis server to take a minute to respond to GET, but let’s figure out what to do on the app side.
Javascript is asynchronous by nature — we live in a world of callbacks. Our code often hides this fact, but application code is in a deep nest of function calls. How do we deal with a situation where one of those callbacks is taking too long?
In our previous installment, I showed how to simulate a slow response with Redis. Admittedly, the method is a bit clumsy but works for the purpose of testing. I used the code from the previous installment to build and test the code here.
My approach to solving this issue is to close over a call to Redis, set a timeout, if the server response comes back in less time then the timeout, then clear the timeout and proceed as normally. If the response takes too long, then we’ll return an err to the callback. The callback will also be re-defined to a no-op to prevent double responses.
This relies on closures, which are often a source of confusion — to compound this we’ll be manipulating the arguments pseudo-array. It isn’t black magic — just stay with me. The description is more complicated than the implementation. Here is my annotated source code:
Fun stuff. So, now you can time limit a specific command to a certain amount of milliseconds. So, you can run something like this:
var shortGet = timeLimited('get',5);//same as client.get('my-string', function(err, value) { ... }
//except it will err out
shortGet('my-string', function(err,value) { ... });
In this example, the script will err out after 5 ms instead of hanging indefinitely.
Here is one caveat: if you are truly experiencing a situation where your Redis instance is busy for an extended length of time, it won’t respond anyway. Remember, Redis is single threaded — so if a particular call is taking 1000 ms to respond, in a single server situation, you’re stuck until it responds— no other commands will be processed. At least with this type of timeout your app layer can send a 5xx Server Error response code or maybe fall back to some other storage instead of hanging.
