The Testing Contract: End-to-end testing strategies for processes that are out of your control
How do you maintain high-level testing when you have neither the resources nor the time to check every system? Can you minimize your risk when you can’t complete testing across the entire process or system?
The answer is yes — by using a testing contract and other strategies to minimize the risk and get as close as possible to end-to-end (E2E) testing.
Before we start, let’s cover a few basics.
What is E2E testing?
E2E testing is a complete approach that takes place from the very first point of contact with the application all the way to the end result. The process checks the quality of the functionalities and communication with external interfaces such as integrated platforms and databases. This is typically the last step in the testing process, doing targeted testing of separate functionality before bringing it all together.
Why should you use E2E testing?
Use E2E testing for any complex process that integrates with external and related systems that rely on each other. If any function fails or is unable to perform the intended operation exactly as planned, the result could be compromised.
Testing contract
Whether formal or informal, a testing contract is crucial for building the required level of trust and to be clear so that everyone understands what they are doing. Once the contract is developed, set up a meeting to kick off the testing process, review responsibilities, and ensure everyone will pull their weight.
The contract should outline the entire testing process and highlight areas of higher risk that will need more attention. Identifying those key areas will minimize that risk and achieve as close as possible to a complete E2E process.
In the use case that follows, we outline three examples of potential areas of higher risk that would be outlined in detail in your testing contract:
1. Zero in on processes that pull in or send data outside of the system (the inputs and the outputs).
2. Review data-retention settings that may affect performance immediately and over time.
3. Focus on steps before data is being sent to and from systems.
To illustrate these strategies, let’s review a use case where internal sales system developers are creating a process that will update an email subscriber status in their sales software.
The solution involves a multi-step process that has been developed by the external email developer team. The sales team doesn’t have the time or the resources for complete testing of the email platform, so they are relying on the external team to do the testing.
Testing strategy
The following testing process has been proposed:
Step 1: The email team will perform E2E testing of the internal and external systems, going through each step of the process. They will address any bugs on the external side and will complete testing and bug fixing once the system checks out.
Step 2: The sales team will perform a unit and vertical-only test in their platform. They don’t have the time or the resources to test the email process, so they will have to rely on the email team’s testing.
Issue: How does the sales team limit their risk by not being able to do E2E testing and rely on the email team to do high-quality testing of the entire process?
Testing contract
Develop the testing contract by outlining the testing strategy and highlighting the risk areas and parts of the process that should receive greater focus when testing.
Here are three strategies for identifying areas that might need greater focus:
- Zero in on processes that pull in or send data outside of the system (the inputs and the outputs).
Let’s take a closer look at step 2 of the email process, which involves transferring files to an external FTP. Whenever an FTP is involved, a red flag should go up as the flow of data requires access that can be denied or delayed for a variety of reasons.
Isolate this step, and work with your external partners to dig into details of the FTP. Some possible questions and areas of focus could include the following:
· Is it a secure and reputable FTP client?
· When do passwords change, who does the update, and how are they communicated?
· Have there been any issues in the past? If possible, review the issue log.
API connections can also be troublesome, so take the time to review specific settings and configurations:
· Are there data limits?
· Will the flow time out depending on usage? The above example has a data flow going out to the external system that timed out if it wasn’t used after 30 days. At that point, a call for another API token was required.
· Are there times of day when traffic can be compromised or slowed down?
· Are there any backup plans to reroute because of a broken connection?
· What are the data recovery options?
2. Review data-retention settings that may affect performance immediately and over time.
Processes that involve writing and storing data should also be closely reviewed. Some areas to consider include the following:
· Understand how the data is stored and what the storage limitations are.
· Will the data be updated or overwritten?
· Are there specific field types and data lengths?
In this use case, data is being written to files called Salesforce data extensions. One of the out-of-the-box features for these data extensions is the retention settings, which will be different depending on the user. One user might turn them off; another might have them all set to delete after 30 days. Understanding the preferred settings and their impact both in the short term and the long term will help ensure success.
Make sure the retention settings are not active in the Salesforce data extensions. In this example, not only the data would be deleted after 30 days but also the data extension itself.
3. Focus on steps before data is being sent to and from systems.
Take a closer look at the steps surrounding the data transfer as those processes might be vital to the success of the overall process.
In the example above, step 4 is the last step in the external process before the data is pulled back into the internal environment. The purpose of this step is to pass the most recent status of a subscriber back into the system. This is the final step, so make sure the inputs and outputs of this process are correct.
When reviewing the final steps, consider the following:
· What is the intended result? Confirm it from both ends.
· How does the data change from step 3 and 4?
· Does the script sending back to the internal environment call on the data source in the final step (Data Extension B), or does it point to another data source (Data Extension A)?
Let’s review the three strategies for minimizing your risk of issues when you can’t do E2E testing across all systems:
1. Zero in on processes that pull in or send data outside of the system (the inputs and the outputs).
2. Review data-retention settings that may affect performance immediately and over time.
3. Focus on steps before data is being sent to and from systems.
Testing isn’t a perfect science, and bugs will always happen. No organization has an unlimited amount of resources to do extensive E2E testing of all systems, and you can only do what your resources allow you to do. Taking the time to spend your resources in the right areas can minimize that risk and give you the optimal outcome each time.
Consider developing a testing contract, take the time to understand the limitations of your environment, focus your energy on those high-risk areas, and gain buy-in from all groups.
About Cervello, a Kearney company
Cervello, a Kearney company is a data and analytics consulting firm and part of Kearney, a leading management consulting firm. We help our leading clients win by offering unique expertise in data and analytics, and in the challenges associated with connecting data. We focus on performance management, customer and supplier relationships, and data monetization and products, serving functions from sales to finance. We are a Salesforce partner and help our clients implement, customize, and optimize the platform into the best solution for their needs.
