The “Hot Duck” Test
Early on as parents, we decided to stock up on bath toys. Apparently thrift stores around here have quite a bit of selection for not much money.
We found this bright yellow rubber duck meant for babies, because it had a pretty cool feature: On the bottom, there was a place where it would say “HOT” in bright white letters if the water was too hot.
Great idea. Except it said “HOT” all the time. Like, even right there in the store where it was a nice comfortable room temperature.
I suspect that was one reason why it was in a thrift store.
But this duck reminds me of a type of automated test that just yells “HOT!” all the time.
And if you weren’t familiar with the test suite you might think something was wrong. That is, until you looked at the actual system (like testing the water with your elbow like our ancestors did) and realized: things are fine.
Or maybe a manager sees the failing test and goes “!” until someone stops and explains that it’s a false alarm.
Tests like that, I think of as “Hot Duck Tests”. And they waste thousands of dollars.
What can we do about these types of tests? Two strategies come to mind:
We can always take some time, dig into the failure, and figure out what’s causing it. Then fix it.
Sometimes you’ll find these types of tests are of another type, which I call “God Tests”. They’re huge and try to hit as much in one pass as possible. A single point of failure in there fails the whole test.
It’s better to break God Tests into smaller tests, which could even be run in parallel and speed up testing.
Sometimes you’ll notice the Hot Duck test relying on data being set that isn’t present anymore, and no one’s had time to go back and reseed the data again.
It may be worth coming up with a way for the tests themselves to seed their own data (if it can be done quickly enough).
If there’s a way to fix it, go for it. But if it’s not possible, or there’s not time, there’s one more thing you can do. It’s super easy:
Not even kidding. Drop that Hot Duck like a Hot Potato, yo.
We measure tests by the amount of value they add. Tests like this aren’t “neutral value”. They actually remove value.
People have to spend time, thought, energy and money on these tests just to be aware that these will fail all the time.
New people will waste much needed onboarding time on something that’s a known issue that everybody else just ignores. Plus it puts less faith in the tests.
Too time consuming to fix? Delete. Better to spend your time elsewhere.
Time is Your Most Valuable Resource
Spending it on the wrong thing will cost you a lot more of it in the long run. What’s the most amount of time you’re willing to spend on a fix? If the estimate to fix is higher than that time, drop the test.