Simulating a slow command with Node, Redis and Lua

This is part 12 of my Node / Redis series. The previous part was Save Node.js headaches with Lua and Redis.

Look — we all know that Redis is super fast, with simple commands executing in less than a single millisecond normally. However, with more complex commands mashed together in MULTI / EXEC blocks, you can end up burning several milliseconds. Not good in a (functionally) single thread environment. Let’s say you want to simulate how your app will perform in a worse-case situation — how do you force Redis to take a huge amount of time to complete a command?

Before we get into exploring this subject, a small disclaimer. What we’re doing to simulate a slow command is totally insane. Do it on your dev machine, but please, don’t do it on a production environment — your CPU will shoot to 100% and your Redis server will be out-of-commission for a short period of time. Don’t borrow any code here and think it is some sort of good idea for any other purpose. This is a terrible thing to do and you shouldn’t do it unless you’re trying to simulate something terrible.

It is [thankfully] hard to make this sort of thing happen with Redis. The most obvious approach would be to load garbage into Redis and do some CPU intensive operations (SDIFF, SORT, ZUNIONSTORE, etc). I didn’t take this approach because the cleanup would be annoying and I feel like the results could be inconsistent. A few commands can simulate blocking behaviour, but not the high CPU load: CLIENT PAUSE, BLPOP, BRPOP, BRPOPLPUSH.

So, that leaves me with Lua. I have mixed feelings about Lua in Redis with Node, but this falls into the right tool / right job category. Natively, Lua doesn’t have even a wait function, but there are ways to make this happen. Interestingly, those ways of making Lua wait are not available to the Lua interpreter in Redis. At any rate, this doesn’t solve the busy problem.

So, let’s break some rules. Lua in Redis will allow you to do silly things like infinitely repeating loops that immediately shoot your CPU sky-high. Let’s do that — but I want a maximum execution time. Without access to TIME, this is a bit of a challenge, but we can kludge it together.

So — my first attempt was to create a temp key, give it an expiration (10ms in this case) and just do a loop for a certain number of cycles or until the key disappears. The cycles allows for a bit of safety vs an infinite loop.

To run this, just pass in the number of cycles you want to devote with your EVAL. Something like this:

redis-cli EVAL “$(cat wait.lua)” 0 1500000

It won’t work — it would always do the maximum number of cycles even though the key should have expired. I’m honestly not sure why EXISTS is always returning a 1 instead of a 0 in this circumstance. I’m guessing it has something to do with Redis not having enough time to do the evection while running the Lua script.

As they say about horses and saddles, let’s try again.

This time I’m going to rely on PTTL itself to determine when I’m done. Basically the same script but instead of checking for the existence of the key, we’ll just check to see if the PTTL reaches 0 (effectively gone).

This seems to work — passing in the same arguments (1,500,000 max cpu cycles) I’m getting a return of 21k-15k cycles on my machine.

Now, let’s bring this all together in Node:

When you run this, you’ll get some troubling results. Remember, we setup our Lua script to stop after 10ms. My results looked something like this:

execution time 22
null 17809

Why is it taking 22ms instead of around 10? What’s going on here is a nuance of node_redis. One of the nice features of node_redis is that you don’t have to wait for a connection event to complete — the library buffers commands until the connection is established. It would be annoying in a non-blocking environment like node to have to contain all your application logic inside a function that is called after the Redis connection is established, so from a syntactical perspective, this is a godsent. However, if we are looking to prove that our function is working appropriately, we need to do something different. Thankfully, the node_redis client object has a connection ready event that allows us to run commands only after the connection has been established (thus side stepping the buffered commands).

Now, we’re getting a bit closer to our 10ms goal. I’m averaging between 12–15ms of execution time, close enough. I’m going to chalk up the difference to interpreting the Lua and returning back the response.

This has allowed me to take a look at a few worse-case performance scenarios- should a particular Redis call be in a middleware? Perhaps in an ajax call after the page has loaded. I take for granted the speed of Redis, but I know there are times where something else is going on that degrades the server— being able to simulate these situations allows for a web app that ‘feels’ fast even when the server is grinding away.