On quality control and continuous delivery

There’s been some chatter that Marcin’s post on how a 25 year old font was accidentally resurrected indicates a failure in quality control. I prefer to see it as a success in continuous delivery.

The idea behind continuous delivery is that you move forward fast, so you can correct quickly. You don’t minimize the likelihood of failure, but the cost of failure. To that end, we have automated tests and processes that probe for errors during the release process, prioritizing identification of critical bugs in core features. For non-critical issues we aim to detect and react quickly, without slowing down or blocking the process.

A good release process is like a Japanese railway: regular, safe, and predictable.

In the case of these fonts gone awry, the changes were manually tested on several Windows version and browser combinations before being merged. And while Windows still has a large market share, it may be interesting to note that less than 25% of our users are on Windows, additionally not all Windows browsers were affected. So while the testing was not exhaustive, the bug didn’t cause any harm and we were able to detect quickly, push out a fix the same day, and then tell a fun story about it.

More exhaustive automated visual testing might have caught this, but it wasn’t actually a regression so quite possibly wouldn’t have been flagged anyway; a change was expected, just not this change.

So I chalk this up as a success. A designer was able to make substantial changes to the production site, push code that affected both the web and mobile apps, and was able to quickly correct the error after it was detected.