Regression Testing — Time Efficiency & Risk Analysis

Optimizing your product regression testing plan

This post is written with manual testing in mind, but can be applicable to automation testing as well.

Whenever you join a new team or inherit a new product, you and your colleagues may find yourselves with a huge regression test cases inventory. It may take you a lot of time to understand the volume of the inventory and relevance of each test case and suite.

First, make sure only the relevant test suites and cases are kept active. You most probably should back up the irrelevant tests (just in case), but taking them out of the active inventory is step one in making some order.

Second, if you can validate that test cases are still valid according to the current product flow & behavior it would be super important. It will save a lot of confusion, because these test cases may be executed by new testers who will not be able to run them and precious time will be wasted, exactly when you don’t have it. So, pro-actively handling them when you want to, will be the smartest thing. Best case scenario should be asking a junior QE to run them as part of his/hers on-boarding process.

Third, look for duplications. You’d be surprised to find that due to different projects and different people, there are almost identical test cases in the inventory, that with little to no work can be unified or dismissed.

Now, that we have only relevant, valid and unique tests in place, you need to examine the current regression testing process. If your team was used to run ALL tests before each release cycle — we need to check:

1. Do we really need to run all the tests?
2. How much time does it take to complete a regression testing cycle?
3. Has this process proved itself and will it be relevant in 1–2 years from now?

Sometimes, just the force of habit is a strong human nature and sometimes, circumstances have changed throughout time. The fact something was done in a certain way before, doesn’t necessarily mean it shouldn’t be changed. So question the process until you’re proven wrong.

Long tests that barely find bugs can and should be fine-tuned. Real customer scenarios and issues we found out at the last moment or even in production must find their way into our regression testing.

Make sure you run the regression final sanity tests on the exact product build that user gets, including set up feature flags, A/B tests and specific customer configuration. In case of a B2B customer, having real customer data as part of testing will be very beneficial (database, models etc.)

If the test inventory is well built, it probably has functionality based sorting. This is super important to maintain, because sometimes you will need to run specific function regression instead of a full regression and even repeat it to support development. Tagging/labeling/annotating the test with the relevant functionality/area can be very useful for both tester and developer. Specific automation tests can be triggered by the build (as part of continuous integration cycle).

If you think you’re running too many regression tests or it takes too much time to run them before each release and the business would like to release more often, then you need to start practicing Risk Analysis.

The release regression plan should be consisted of:

1. Tests that must be included in each plan: high level acceptance tests, final sanity, important customer scenarios.
2. Tests that are needed for this specific plan: tests of new features and new bug fixes (you should be able to know all these beforehand).

New features should have been tested thoroughly before being merged to the main development branch, so you probably don’t need to run all their certification tests as part of the regression and going forward.

Another way to look at regression testing risk analysis is to simply examine the number of tests included in the regression plan.

You can set a threshold, a limit of tests per plan (assuming most tests run at similar durations).

For example, you can decide each regression plan should not surpass 200 test cases. In this case, you force yourself and your team to plan accordingly, make the educated prioritization as part of plan building and keep up the time frame.