End to end or Contract-based testing
For the past few years a debate has raged over whether contract-based testing can replace functional, or end-to-end, testing. Some suggest substituting end-user-focused testing with unit-test frameworks, written and executed during code deployments.
Is it possible to completely substitute end-to-end functional testing with a suite of unit tests routinely checking integration points? Maybe.
But before you decide to replace functional end-to-end testing using contract-based testing, you’ll need to identify where they are similar and where they’re not.
End-to-end testing
End-to-end testing involves testing a user workflow from beginning to end, including all back-end processing engines, be they API services or other messaging or data transfer services. This includes testing an Entire application from the beginning of the user-workflow to the end.
Contract-based testing
Consumer-based contract-based testing is exactly what it claims to be:
- You test contract agreements between a consumer endpoint and an API provider endpoint.
- Contract-based tests are designed to cover each interaction scenario that takes place between the user and provider endpoints.
- You’re creating unit tests that validate that your API connections are functioning as expected at any given time and according to your requirements in contract with Users.
Interfaces that cannot be used directly because they are “production” endpoints can be mocked or faked and still tested.
Contract-based test execution explains you that the API endpoints:
- Works as expected.
- which endpoints that are up and active.
- it may have broken a previously working connection on either the consumer or provider end because of changes made in API structure/parameters.
Considering contract-based tests won’t detect endpoint configuration issues or misused classes, other types of testing like integration of workflow test cases cover those errors and catch those types of defects.
One can’t substitute for the other
Contract-based unit tests only check that API endpoint connections are active and functioning correctly. But that’s not all the testing the application requires.
It’s highly unlikely that contract-based unit tests have checked any consumer workflows or consumer-facing functionalities. However, there are certain ways to mix the testing types you require to make sure the quality of your application before releasing it, for best testing coverage, consolidate them.
Can you replace automated API test executions? Possibly. If the provider changes its code and it breaks the application then contract-based tests can find those issues, but it may not be able to identify you where the break-point is, however, they include information to help you locate the breaking point.
So, before you substitute API-based functional tests with contract-based tests, ensure that the tests they execute are clone of each other.
Consider keeping tests that are not cloned, so they catch issues that might not be covered by the others.
Contract-based: How much coverage do we need?
Contract-based tests also require contact between provider and consumer for test execution.They must be tested against a mocked system and not against the production instance of an external service.
If it’s necessary to check the tests on the production instance, contact the provider before hitting its endpoint with tests. You’ll both need to be aware of the impact of your tests and what, if any, data is added or changed.
Another consideration with contract-based testing, as with nearly any other type of testing, is depth. How much test coverage do we need?
Contract-based testing doesn’t cover the full functionality and only scratches the surface. Following the full functional paths tends to lead the test developer down to the potential rabbit holes, resulting in tests that break frequently and are hard to maintain.
Use contract-based tests to focus on catching defects:
- Within the consumer workflows
- Around misunderstandings regarding endpoint configuration or payload
- When the provider has created breaking changes on endpoints and/or payload
Get the full workflow
End-to-end testing covers the full end-user workflow from beginning to end, regardless of which back-end systems you’re crossing or using.
End-to-end testing focuses on the end-user workflow functionality and not the particular interface doing the processing behind it.
Frequently, you may find issues or defects that are not required to be logical and are present in the UI workflow.
Remember, unit tests cannot find non-logical defects, nor they detect problems with database communications.For example, if you execute end-to-end tests, you’ll know quickly — when tests continue to fail — that the database is not communicating.
End-to-end tests are to be thorough and complex, which makes them time-intensive & sensitive. However, they tend to find significant issues that otherwise go undetected.
Contract-based testing, on the other hand, is less intensive and can be done more frequently, providing extensive smoke testing for coverage each time code is deployed.
Use both with minimal overlap
Your best overall bet is to determine what functional API or other microservices testing can be effectively tested with contract-based unit tests so you’re not duplicating testing efforts. For example, end-to-end functional testing will uncover misconfigurations that appear in resulting data or the user interface. Code reviews are useful for uncovering nonstandard class and object development.
Contract-based testing cannot replace functional end-to-end testing, but you can combine the two to improve testing coverage across the board.
