The Servers Are Burning
An engineer pushes a small change and OkCupid goes up in flames
By Dale Markowitz
One of the first things I learned after learning how to write software was how to write tests for that software. First you wrote code to perform a certain task — say, find the square root of a number. Then you wrote more code to test whether the first piece of code did what you wanted. Does the function return the correct value? Is two the square root of four?
I thought this type of testing was ridiculous. If you wrote buggy software, why would the software you wrote to check that software be any less buggy? That happened to me a lot: I’d spend twenty minutes trying to figure out why my tests said my programs were broken only to realize that the tests themselves were broken.
Yet what I found even more troubling was that in order to write effective tests, a programmer had to know all of the ways that a piece of software could fail in order to write tests for those cases. If she forgot the square root of -1 was undefined, she’d never write a test for it. But, obviously, if you knew where your bugs were, you’d just fix them. Errors typically hide in the places we forget to look.
My first employer, the online dating site OkCupid, didn’t harp on testing either. In part, this was…
