For a software project to succeed you need a successful strategy to testing. One way of doing this is to use the Testing Pyramid. Whether you are starting out a new software development project or working on an existing project, it is important to have a test strategy in place.
We all know it’s bad to not have any test at all but it is also bad to have useless tests. Partial tests, intermittent tests that fail for no reason, hard to maintain tests are just some examples of bad dev testing. We all know there’s different tests to put in place. For example, unit tests, integration tests, end-to-end tests, UI tests, API tests, component tests, service tests, system tests, and the list goes on.
So you start to put in tests for your code. But where do you start? How do you tackle it?
This is where a Test Strategy comes in.
In this strategy it is best to utilise the Testing Pyramid to help you define and categorise the different types of tests.
A testing pyramid is a pyramid of where all the different types of tests fits. The basic pyramid looks like this:
Very simple. 3 levels.
- At the bottom, is Unit Tests. These should count for the majority of tests you have for your codebase. They tests the smallest unit of code possible. Normally, only test a single method each.
- Middle level consists of Integration Tests which are tests designed to verify the integration of different parts of separate components of a software system together. This can be integration with a Database, with a Framework, with third party external software systems, or even with inter-software components between different layers (if you’re implementing a multi-layered software architecture software)
- At the top is the end-to-end tests which are tests that verifies the end to end workflows of your codebase. They tests the system from the user-action entry point right to the end of the system down to the database level. However, these are typically black-box tests.
A more detailed Testing Pyramid may look like this:
At the same level as Integration Tests you may have what is called Component Tests along side it. These are tests which tests the different components of a software system. Of course, you need to define within your system what constitutes your ‘Components’. This assumes your software is a modular distributed software system with separate ‘Components’, SOA, or Microservices.
Sitting in its own level and in-between the top 2 levels is API tests which are by the name of it tests the API endpoints of your system. Assuming you have APIs of any sort, be it RESTful or non-RESTful. These tests are common because a lot of web apps these days have so-called backend web services which expose its functionality (CRUD like operations) via endpoints. So it is important to test the invoking of these APIs as well. Some people argue these are just Integration Tests. And they’re probably right, they’re merely just a more specific classification of Integration Tests.
Perhaps not so well known until recent years are these tests which tests the ‘UI’ aspect of your system (if your system has an UI). If you’re system is just a backend web service exposing APIs then obviously you don’t really need these and they’re likely not valid. i.e. You don’t have an UI!
At the very top of the pyramid now sits a cloud or bubble of ‘Manual Tests’. These are tests which is performed manually. They test the whole system in a black box way. They are important as they test the system as a whole which each of the individual area tests cannot do.
Who should do what?
Knowing which tests to write is a good start. Next step is probably agree on who is responsible for what.
In a team, you have developers, QAs, DITs, and maybe even SDETs. For a team to function properly you probably need clear responsibilities assigned.
- QAs are obviously responsible for manual tests. Sometimes, SDETs do these tests too. They perform them alongside QAs.
- DITs would tackle the top level of UI & E2E tests. This is what normally people classified them as test automation. Automating tests which automate the testing of the whole system. One way to see them is to think of them as the automatic equivalent of manual tests but one must not think they can replace manual tests completely.
- Developer Tests are basically tests that is deep in the code itself. From the bottom level of Unit Tests right up to API tests. Sometimes, SDETs work alongside Developers to share this responsibility.
- a final word on SDET is that although they normally perform the tests of a QA & DIT together (because they basically are QA + DIT), sometimes more often than not, they work closely with the Developer to share his responsibility too. It is not uncommon to see SDET write tests for the whole Pyramid…. Because, at the end of the day, a SDET is by the name suggests — a Software Developer doing Testing.
Avoid the Ice-Cream Cone Anti-Pattern
Don’t do this.
This is bad. This basically is the inverse of the Testing Pyramid. It is the opposite of what the Testing Pyramid suppose to achieve.
You won’t achieve much with this approach. You could think of this as a bad strategy. Having a big portion of manual testing compared to that of Unit Tests cries for disaster. Development time increases. As it takes much longer to do things manually than having an automated test suite at hand which you can easily run within a small time frame (if you have fast tests that is).
One of my previous company i worked for, we had this anti-pattern in place. I didn’t notice it at first until i left that company only later on to come across a blog post which mentions that. Only then i realise… I’d wish i pointed that anti-pattern out to my former employers earlier so we could have saved a lot of time in development which would have resulted in quicker time to market for our business features.
We had little to none Unit Tests let alone Integration Tests. We did however had a large number of E2E tests with a bit of API tests. But mostly our testing was done by hand manually.
So the bottomline is… avoid this as much as possible.
- Write Unit Tests (this is MUST!, and it’s an absolute basic requirement!)
- Write Integration Tests — integrate with the Database, Framework, 3rd Party external systems etc. Most people treat these as optional and i can’t blame them but i suggest put them in place. They will eventually save a lot of pain and you’ll thank them later.
- If you have APIs, write API tests.
- If you have an UI, write UI tests, otherwise no need.
- Automate as much of the testing you can via the different tests of the Pyramid.
- Don’t think automation can replace Manual Tests entirely. You’ll best to still keep doing these.