Approach to Testing
At ClearScore we build iteratively, focusing on and solving real problems, being careful not to waste time on issues that may never exist. This is a tough balance to find, but when we get it right we are able to to adapt and release updates insanely fast. This does mean we need to have an appropriate testing strategy that can keep up.
In the beginning, traditional unit testing was fine, communication was easy and everyone knew most of the code and how it worked. As our numbers and code grew, it got harder and harder to keep out bugs.
In order to scale effectively, we had to change our approach to testing; unit tests seemed less reliable and required more effort. The first thing we did was identify what it was our approach to testing had to achieve.
What do I want from my tests?
I want a fast feedback cycle. My tests must tell me sooner, not later, if I break anything. Conversely, I don’t want tests to fail if nothing is broken.
I want all coders to feel they can refactor and improve code. I want them to have the confidence to implement changes and to know that if they break something, the tests will tell them.
I want to be able to upgrade any dependency at any time and know if the update affects my app. I want my tests to tell me if any 3rd party library no longer works within my code.
I want for my test to tell me if I am using lib-x correctly.
Traditional — Testing Pyramid
Traditional thinking says we should have most of our tests as Unit tests (ensuring components work in isolation) as these are the fastest and cheapest to run. Moving onto integration and finally end-to-end tests.
The problem we’ve found is that with so many components (700+) we couldn’t rely on unit testing to tell us when something did go wrong.
Long Feedback Cycle
This sounds counter-intuitive, we know unit-test are the fastest way to run tests. Unfortunately for us, Unit Tests didn’t scale well as it took too long to find bugs. Often, when important parts of the code wouldn’t work, we wouldn’t find out until the integration or the e2e tests — and that is if we remembered to test it elsewhere!
This was the biggest feedback I heard when trying to transform how we test, this test doesn’t belong here — it should be done at the e2e level. Testing technology is now so good that we no longer have to wait. We can now write tests at the unit level that typically could only be written at browser level. Doing this keeps the feedback cycle much faster.
Testing Implementation Detail
When refactoring, unit tests would fail even if nothing is broken. This happens when testing implementation details. This isn’t always a big deal if the re-work needed is small. But, when making changes to many components, or if the person making the change lacks context, this re-work can be slow.
To scale when we refactor, we need our tests to fail, and only fail, if we’ve broken something. We do this by not testing implementation detail. i.e. within Redux do not test mapStateToProps.
Not Testing With Our Dependencies
Unit testing would code would mean that we often mock our internal or external dependencies. This meant we often wouldn’t know if we were actually using the dependency correctly and we would test how we thought things worked. What’s worse is that when the tests (incorrectly) passed, more code and more tests, were added to ‘fix’ the problem. This created confusion, misunderstandings and complex code, half of which wasn’t doing anything.
New — Testing Trophy
The testing Trophy instantly puts the emphasis on integration testing. We haven’t got rid of unit testing, and we still say that it is the right solution if you have tough units of code to write. The increase in the use of integration testing though has solved a number of the previously identified problems.
Short Feedback Cycle
We now test multiple components working together at the same. We do not mock any integrations and we do not always unit test individual functions. This may be slower, but it is only slightly noticeable. But in the grand scheme of things, the feedback is quicker as we don’t have to wait to find out if our components work together.
Testing via the API
We only test the function/component using the public API. This means we don’t rely on implementation details to test against. The benefit this brings is that we can now refactor our code, and we only have to change our tests if we break something. The side-benefit is that people now think about how the component is used and think of more edge-cases and unhappy paths than previously.
Embracing dependencies
Typically you would expect to mock dependency, but we actively encourage developers to ensure the dependencies are intact during the tests. Clearly this is not the unit-test way of doing things. The benefits here though are massive. For example, having chosen not to mock our 3rd party library, when there is a major version bump of that software, I know instantly if we can update when all our tests pass. In the old world, we would have to be very careful and rely on our browser tests.
Further Reading
To follow up from this I recommend you read blog posts by Kent C Dodds, and in particular Why I never Shallow Render was eye opening for me.
I’ve now also written up How We Test, which goes into more technical details
