Why You Shouldn’t Test With Thread.sleep()
Why is Testing with Thread.sleep() A Bad Idea? And What Can You Do Instead?
If you are a Java developer, chances are you’ve hit this scenario before: you need to test asynchronous code. So you create a new test case, call the asynchronous code and then… here the problems begin. You need to wait until you actually get the result to put assertions on it. You may be thinking just sleep for a bit by calling Thread.sleep() and all will be good. This article will show you, why this is not a good idea and what tools can be used to the rescue. All code is also available on GitHub.
Passing tests in your sleep?
That sounds too good to be true. And unfortunately, it is. Unless you’re the only developer on your project and you’re not using CI/CD, the tests have to pass in all environments. “Works on my machine” is not enough. So you need to sleep for a long time to make sure that the result is really available in all cases. Otherwise, you end up with a flaky test and annoyed co-workers whose PRs fail for random reasons. Waiting for a long time on the other hand is also not a good idea. It will make your tests slow and over time all these Thread.sleep()s will add up.
To illustrate the issues, we create a simple class that simulates asynchronous behaviour. Handling of some corner cases is skipped for brevity:
public final class AsynchronousOperation {
private final Duration duration;
private Instant completesAt = null;
public AsynchronousOperation(Duration duration) { // 1
this.duration = duration;
}
public void run() {
completesAt = Instant.now().plus(duration); // 2
}
public boolean isCompleted() {
return Instant.now().isAfter(completesAt); // 3
}
}When a new instance of AsynchronousOperation is created, the duration specifies how long the operation should run (1). Doing so gives us the flexibility to change the behaviour when testing and simulating different environment conditions. When we run the operation, we calculate the instant, when the operation will complete (2). Finally, when we check for completion, we simply check if the current time is after the calculated completion time (3).
Next, we create a test case using sleep:
@ParameterizedTest(name = "OperationDelay: {0}ms")
@ValueSource(longs = {50, 60, 40, 200})
void runAsynchronousOperationWithSleep(long delay)
throws InterruptedException {
final var asyncOperation =
new AsynchronousOperation(Duration.ofMillis(delay));
asyncOperation.run(); // 1
assertFalse(asyncOperation.isCompleted()); // 2
Thread.sleep(250); // 3
assertTrue(asyncOperation.isCompleted()); // 4
}The test is parameterized with the operation duration as a parameter. That means this one test case will be executed four times but with different delays. This simulates jitter in the real world, like network delays and so on. First, we create a new operation with the specified delay and run it (1). Right after starting, the run shouldn’t be completed yet, so we assert that it's not (2). Next, we sleep. We are forced to do so for more than 200ms to make sure that even the worst-possible delay passes the test. Hence we pick 250ms. Finally, we assert that the operation is completed.
Running this test in IntelliJ, we see that four test cases are executed. Each of them takes slightly more than 250ms to run, indicating that most of the time is spent in Thread.sleep():
Running only four test cases just took more than a second! Imagine hundreds of tests having to wait like that and you’ll quickly see that this doesn’t scale.
How to fix it: What are you waiting for?
But wait, don’t most of those function calls return much faster? Why do we wait for 250ms if we have the result after 40ms already? For most asynchronous code, we don’t have a guarantee for how long it’s going to execute. However, what we can usually do very easily is check, whether the execution is complete.
This is where awaitility comes into play. Awaitility lets you check frequently if the execution is complete and when it is, continue running right away. Here's what the updated test case looks like:
@ParameterizedTest(name = "OperationDelay: {0}ms")
@ValueSource(longs = {50, 60, 40, 200})
void runAsynchronousOperationWithAwaitility(long delay) {
final var asyncOperation =
new AsynchronousOperation(Duration.ofMillis(delay));
asyncOperation.run();
assertFalse(asyncOperation.isCompleted());
await("completion of task")
.atMost(Duration.ofMillis(250)) // 1
.pollInterval(Duration.ofMillis(10)) // 2
.until(asyncOperation::isCompleted); // 3
}We create and run the same async operations as before. However, instead of sleeping, we now await the operation to be complete. We set a maximum delay of 500ms (1). If more than this time passes, we will get a ConditionTimeoutException which fails the test. After all, we need to make sure that we don’t wait forever. Next, we specify to poll the status of the operation every 10ms (2). Finally, we specify the condition we’re waiting for, that asyncOperation completes and the corresponding function returns true (3).
The improvement in timing is huge:
The test cases went from running ~250ms each to 47ms, 53ms, 63ms and 213ms. A total of 376ms, which is almost 700ms or 66% faster than the original run! Why? Because we frequently checked if we still needed to wait and stopped doing so, right when we had a result.
We also increased the stability of the test for free. By waiting for up to 500ms, we can now catch outliers with a very long execution time without impacting the test runtime even more.
Conclusion
By using a tool like awaitility, you can make your tests faster and more reliable. This post gives a basic introduction to the tool. For more details, check out the official usage guide.
If you want to keep getting updates about similar topics, leave a subscription and share with friends and co-workers :)
David
