An Undogmatic Approach to Testing
Testing might simultaneously be the least understood and most discussed subject in software development. Some consider it an utter waste of time and others can’t fathom a software development cycle without it.
Does the following sound familiar to you?
I did not write tests because I did not have the time.
I did not write tests because the client did not have the budget for it.
I did not write tests because I needed to get it up and running right away, and once that was done, what was the point of writing tests anymore??
I did not write tests because the business logic was not clear in the beginning.
And my favorite one:
I could not write tests because the existing code was not testable.
Even if you write tests, the conundrums continue:
Should we write tests first?
What does TDD really mean?
Which percentage of tests should be unit tests?
Which percentage of tests should be functional tests?
It cannot be my code because I wrote unit tests.The tests passed so everything should be fine.
I’m not done with the ticket because I have been writing tests for the last 2 weeks.
There is an interesting debate between two renowned computer scientists, Jim Coplien and Robert Martin, about Test Driven Development (TDD). Both have published several bestselling books; Jim Coplien considers TDD to be voodoo, while Uncle Bob swears by it.
Even those who consider testing to be a necessity cannot agree on some fundamental matters. Should all interactions with a database be mocked? What does a ‘functional test’ really mean? What qualifies as ‘well-tested software’ and how do we measure such a thing? What does code coverage really mean? What percentage of code coverage is too much, and what should be considered a minimum? Should every single pull request contain tests? If yes, what types of tests? As of today, there is no consensus, and no clear evidence to support any one doctrine. Over the course of my career, I have been fortunate enough to work on a variety of code bases, ranging from ones which barely had any automated tests, to others which had thousands. I have seen both extremes in successful companies with healthy profit margins and happy customers.
Is computer science really a science? If so, then why can’t we agree on these simple things? Or maybe it is time to draw a clearer distinction between computer science and software development, unlike the former, software development might be closer to an art.
So in the case of testing, which approach is the right one? In my opinion, it depends. Before you lose your temper for having read this far for such a clichéd answer, let me elaborate…
I firmly believe that almost every solution is a trade-off, even those that don’t seem to have any drawbacks. This applies just as well to automated testing. While it can save you thousands of hours or even millions of dollars, it could also wastefully expend equally large amounts of time and resources. The trick is to understand the context in which your testing strategy needs to function.
Let us consider the case of data intensive web applications, the kind which rely on a database and its contents as their primary source of value. When working on such applications, my testing strategy is as follows:
- When a new bug appears, I write a new test, a functional test, and I do not shy away from having it hit the database.
- When I need to refactor legacy code, I first write as many functional tests that seem necessary, after which I proceed to thoroughly refactor everything with peace of mind.
- When I see a complicated function that I don’t understand at first glance, I write tests with multiple practical cases, such that my input-output mapping for the test cases double as documentation.
- When I work on a new feature, I write functional tests for the highest layer of the application, oftentimes the controller, as opposed to the lowest layer. Once the feature is complete, I then add some unit tests for internal components.
At this point, you might be wondering why I’m writing more functional tests than unit tests. Does that not go against the sacred Testing Pyramid? You might also find it disconcerting that my tests heavily interact with the database. To revisit my original argument, you have to consider the context.
In a perfect world, even data intensive applications would benefit from using the testing pyramid, but this would require well managed state, minimal coupling, idempotent functions, proper separation of concerns, and an ability to mock all input-output dependencies in a realistic manner. If these are starting to sound a lot like the many excuses for not writing tests that I brought up earlier, it’s because there is some underlying truth and validity to all such excuses. In the real world, we often do inherit code bases that require us to improvise our testing strategies. If a unit of code is not independently testable, writing unit tests for it would be counter-productive. While refactoring such code just to make it testable as a unit may introduce new bugs or silently change the behaviour of your application.
Therefore, in this situation, as per my best judgement it makes sense to write more functional tests than unit tests, and to create tests that replicate real-world database interactions. If the situation allows it, I use an in-memory database like SQLite, but if that does not accurately represent the constraints and challenges of the production environment, I have no qualms about using a bulkier alternative. Prioritizing functional tests allows me to modify lower level systems more confidently because my tests, if structured correctly, can guarantee that I don’t modify any existing high-level behavior of the application. The basic rationale behind my approach is to favor pragmatism over dogmatism.
So, does this work for every single application?
Of course not!
If you are working on an application that uses a lot of mathematical functions, you’d expect your functions to be idempotent, fully justifying a testing strategy that relies primarily on unit tests with a comprehensive range of static inputs for all tests.
In the case of most web-based applications however, input-output with external data sources tends to be simultaneously the most important and most complicated part to test. If the health and integrity of your application is reliant on its interactions with dynamic data sources, writing unit tests with large amounts of carefully mocked data and no interactions with the data sources can leave you with a false sense of security.
Another fallacy with unit tests is that of success equating to 100% code coverage. Having 100% code coverage does not guarantee bug-free code. It only guarantees that your code works within the constraints under which it was tested. It is often impossible to test for all edge cases. Rather than treating the rate of code coverage as a numeric target, it’s important to consider the nature of the coverage and ensure that the constraints defined in your tests cover most situations of real-world relevance. One practical strategy is to add a new test for every new bug the application encounters. By adopting this strategy, you accept that your code coverage may never be perfect, but with every new bug, the code-base will become a little more stable.
Editorial reviews by Deanna Chow, Carlo Gentile, Nebez Briefkani, Liela Touré & Prateek Sanyal
Want to work with us? Click here to see all open positions at SSENSE!
