Enough testing
What kind of tests to write? What test coverage to aim for? Should tests be written before code? These are some of the many questions around software testing that are widely debated, and with some strong opinions. I can’t possibly do justice to all those discussions by making a conclusion in a few paragraphs, so, I’d rather share practical examples of instances from my work when employing a certain testing strategy greatly benefited the project/system.
I’ll start by admitting that I didn’t use to write tests before code, especially when building a pet project or something for my personal use. But — as I’ve found out — tests before code (aka TDD) is a savior and I can’t stress it enough, as you’ll see in the examples below. Also, spoiler alert: I don’t believe in always striving for n% test coverage (I appreciate though, that there might be some critical systems where a high test coverage is important, for various reasons).
Me likes TDD
The first example is from my early days when I was working on a schema generator for a data pipeline. It started out as a single Python file and I was very happy to add some unit tests and call it a day. It wasn’t long before it transformed into a pipeline of its own, covering everything from schema authoring to schema publication and in between. That’s when the schema transformation part became so complicated that it became hard to remember all the happy cases, let alone the corner cases. And, people started authoring unimaginable schemas (as people do), causing the schema transformer to spit out weird errors. So, I gave up and had my first taste of TDD, as every time an error was reported I’d add a test case for it (even if it was an expected error) and make adjustments to the transformer, if need be. It wasn’t a pretty adoption of TDD, but it taught me a lesson: writing code to pass tests is easier than writing tests to cover all paths of code.
Acceptance testing to the rescue
The second example is from my middle ages (a bad attempt at an AoE II reference), when I was working on a data aggregator for an e-commerce product page. This thing was scantily tested by some unit tests, and hence, errors popped up every now and then, requiring bug fixes during feature development for unrelated parts of the code. After some retrospection in the team, we decided to adopt acceptance tests and make them compulsory for all new features and bug fixes. This helped in two ways: slowly all paths of the code became tested, and regressions started to become rare. There wasn’t necessarily an agreement on test before/after code, but the lesson I learned here was: just as a single line of defense is not a good strategy, a single layer of tests can’t provide the same guarantees as multiple layers of tests.
The unison: ATDD
The third example is perhaps a culmination of the previous two; it was not too distant in the past, when I was working on a sale price calculation pipeline. This thing was being built in a fashion, that could best be described as a test-boilerplate-paths-only i.e. tests existed for only the boilerplate or standard paths of the code such as the HTTP client etc. The end result was that, while the microservices of the pipeline ran fine, they spit out numbers that were nowhere close to the legacy calculations. And, further development of those services made it harder to debug and fix bugs, all the while the team had resigned to accept regressions as part and parcel of the development cycle. Covering all critical paths with a suite of unit tests was impractical, and so, any discussion on improving the testing had to start by throwing the test coverage out of the window. We had some discussions in the team and I proposed that we try out ATDD (the union of acceptance testing and TDD). So, every new task had to be accompanied with (and, started by) writing an acceptance test with acceptance criteria based on the legacy calculations. There was this initial phase, where we had to introduce a bunch of acceptance tests to all the services, but soon after it became possible to adjust existing acceptance tests in case a new feature was not being added. This markedly reduced the regressions and finally resulted in development cycles that were not interrupted by bug fixes. The lesson I learned here was: reaching high test coverage can be a fantasy — sometimes, so choose the type of tests that can deliver the most value.
In a nutshell, my experiences have taught me that, making tests a first class citizen of the codebase and having diversity in the types of tests will save you from pulling your hair out!
