Mirror mirror on the wall, which scenario should I test first? -P1

Part I — History Based

While time is the most precious thing in the world, are you testing without a plan?

Let’s face it, time is the most important treasure in this life. Even if you are the richest person in the world, you cannot turn back time, slow it down, or speed it up.

As a software quality engineer, throughout the years and experiences, I have spent in this field, the moments when I felt the most nervous were in the limited time after the deployment was completed. In this limited time, the QA team is expected to test quickly and provide feedback. Being smart in this limited time and testing test scenarios within the framework of a plan will undoubtedly increase efficiency.

It is very important to use time effectively not only in the remaining time after deployment but also in limited days during the sprint period. Time is money and the QA team must use this time efficiently.

In this limited time, the QA team is expected to test quickly and provide feedback.

This is where software Test Case Prioritization (TCP) should come into play. Software test scenarios should not be tested randomly but should be tested in a certain order and prioritization.

Test Case Prioritization (TCP) is recognized as the most effective regression technique for addressing these challenges by reordering the test cases that encompass numerous altered elements of a product variant to meet the desired criteria. [1]

In general, test case prioritization approaches schedule test cases in an order so that test cases with a higher priority will be executed earlier.[2]

The following approaches are generally adopted in test case prioritization work: [3]

1. Search-based
2. Coverage-based
3. Fault-based
4. Requirement-based
5. History-based
6. Risk-based
7. Cost-effective-based
8. Nature-inspired-based *

In this article set, we’ll investigate:

• History-Based =>Part I
• Risk-Based =>Part II
• Cost-Effective-Based =>Part III

We’ll also deep-dive into the nature-inspired meta-heuristic approach in Part IV, which is also my master’s thesis topic. However, let’s focus on the other approaches first.

1. History-Based Test Case Prioritization:

History-based test prioritization approaches prioritize test cases based on historical execution data. [2] Historical data comprises diverse information types, including details on test case failures during specific test sessions, modified files, and the duration taken for the execution of each test case. Kim and Porter leveraged the execution history and fault detection history to determine and prioritize test cases. [4][5]

In the literature, there are some different approaches to prioritizing the test cases by historical data. ROCKET and AFSAC are some of them. The ROCKET approach’s characteristics of each test case and project represent the probability or pattern of recurring failures. Then, you can utilize the analyzed data to establish the weight values for prioritization purposes. [4]

Get to the point…

Let’s assume that, we have 20 test cases in our test suite of a web e-commerce application. During this execution, we record the outcomes of each test case:

• Test cases 1, 3, 5, 8, 10, 12, 15, and 18 uncover defects or issues in the application.
• Test cases 2, 4, 6, 7, 9, 11, 13, 14, 16, 17, 19, and 20 pass without any issues.

Based on the outcomes of the initial test execution, we can prioritize the test cases for future regression testing using a history-based algorithm.

• Priority 1 (High): Test cases that uncovered defects in the previous execution are given the highest priority. In this case, test cases 1, 3, 5, 8, 10, 12, 15, and 18 would be prioritized first since they have a history of revealing issues.
• Priority 2 (Medium): Test cases that passed in the previous execution but have a history of uncovering defects in previous executions might be given medium priority. However, since none of the passing test cases have such a history in this example, we may skip this priority level.
• Priority 3 (Low): Test cases that consistently pass without uncovering defects may be given a lower priority. In this case, test cases 2, 4, 6, 7, 9, 11, 13, 14, 16, 17, 19, and 20 would be assigned a lower priority for subsequent executions.

For the next regression test cycle, the test cases would be executed in the order of their priorities. This means that test cases 1, 3, 5, 8, 10, 12, 15, and 18 would be executed first to verify if the previously identified defects have been fixed. Then, the remaining test cases would be executed in their prioritized order.

OK, but it’s is already predictable. Say something more logical!

This approach, in my opinion, is one of the easiest ways if you want to prioritize the test cases by historical order, however, it is not an academic or the most logical approach.

Let’s get wider the approach on one more level.

In this example, we’ll assume that we have historical data for the outcomes of each test case over multiple test executions. After getting this data; we can create a probability matrix like this:

The values represent the probability of each outcome based on historical data. For example, if a test case has a pass probability of 0.25, it means that historically, it passed 25% of the time.

Using this probability matrix, we can prioritize the test cases as follows:

1. Priority Calculation: For each test case, we calculate its priority based on the probability of failure. The higher the probability of failure, the higher the priority.
2. Priority Order: We then order the test cases based on their calculated priorities, from highest to lowest.
3. Subsequent Test Execution: Test cases are executed in the order of their priorities, with higher-priority test cases being executed first.

In this example, test case 3 has the highest probability of failure (0.80), so it would be prioritized first. Test cases 1, 2, and 4 have lower probabilities of failure and would be prioritized accordingly.

This approach aligns with research methodologies and provides a systematic way to prioritize test cases in software testing.

You can also categorize the probabilities:

• Test case 3 has the highest probability of failure (0.80), so it receives a high priority.
• Test cases 1 and 4 have moderately high and low probabilities of failure, respectively, so they are prioritized accordingly.
• Test case 2 has a probability of failure of 0.00, indicating that it rarely fails, so it receives a low priority.

We’ll focus on the Risk-Based approach in part II of this article.

Happy testing!

References:

[1] Sahak, M., Halim, S. A., Jawawi, D. N. A., & Isa, M. A. (2017). Evaluation of Software Product Line Test Case Prioritization Technique.

[2] Huang, Y. C., Peng, K. L., & Huang, C. Y. (2012). A history-based cost-cognizant test case prioritization technique in regression testing. Journal of Systems and Software, 85(3). https://doi.org/10.1016/j.jss.2011.09.063

[3] Shah SAA, Bukhari SSA, Humayun M, Jhanjhi NZ, and Abbas SF (2019). Test case generation using unified modeling language. In the International Conference on Computer and Information Sciences, IEEE, Sakaka, Saudi Arabia: 1–6. https://doi.org/10.1109/ICCISci.2019.8716480

[4] Cho, Y., Kim, J., & Lee, E. (2016). History-based test case prioritization for failure information. Proceedings — Asia-Pacific Software Engineering Conference, APSEC, 0. https://doi.org/10.1109/APSEC.2016.066

[5] J. M. Kim and A. Porter, “A history-based test prioritization technique for regression testing in resource constrained environments”, Software Engineering 2002. ICSE 2002. Proceedings of the 24rd International Conference on, pp. 119–129, 2002, May.