The Right Mindset About Defects and How We Can Reduce It to More Than 50% in a Quarter

Most of us must have heard of defects. Especially in the world of software development, product and defect seem inseparable. In almost every feature development process, defects are introduced. Rarely do we hear of a development process with zero defects. Even when there are zero defects, those of us involved in the development process may not believe it and wonder why this could happen. It might even be a concern whether there is something missing from the testing process that has been done. Is there anything that seems to be skipped? At the end of this article, we will know if these worries and questions can be avoided after we know some of the mindsets and concepts that we need to pay attention to when dealing with defects.

Fewer defects are important, but the effort to find high-quality defects is more important. Then what does the high-quality defect mean? It is a defect that appears when we have done or are aware of the following things:

1. Our ability to prove that we have done a deep exploration of a feature.
2. Digging and finding various extraordinary edge cases.
3. Ensure that the features or flows that will be released provide benefits to customers, do not interfere with the customer experience, or even harm the customer.

Here are some things we can do in order to find (also prevent) high-quality defects:

1. Identify potential defects as early as possible. For example, participating in the discussion process related to PRD (Product Requirements Document). Understand the details of the product to be developed including but not limited to the description, background, objectives, target users, scope, related features, and user flow sections. Build a deep understanding of our targeted users, and work closely with the product team to help improve existing PRDs. In this case, we can give a challenge if we feel that the feature that will be developed somehow has some things that have the potential to disrupt the customer experience or can actually harm the customer.
2. Provide special time to explore and think about various possible test cases according to the product to be developed. In this case, we can discuss together with all team members (PM, engineer) to get various points of view. The output of this discussion is in the form of testing notes.
3. From the list of test cases, brainstormed with the engineer to determine a test case will need to be executed at which level of testing (unit test, fragment test, API test, or UI end-to-end test). In this brainstorming process there are several questions that we might be able to ask: How we can test the features? Can we cover by isolated test (unit test and/or fragment test)? How about the integration test itself? Are we confident without it? Do we still need a UI end-to-end test for a safety net? Is there any test case that we haven’t covered yet?
4. Along with the increasing visibility of the products made because the development process has started, the possibility to find various other test cases is getting bigger, so that we can complete the initial test cases that have been made. The exploratory test here is important to be able to do.
5. In the end, it is also necessary to carry out a user acceptance testing process with PM.

So the fewer defect mindset should always be accompanied by extra effort beforehand to explore, find, and prevent high-quality defects.

What is happening now is that some software development teams are only aware and focused on the number of defects. They are not aware of the quality of the defect that occurs. They were quite happy when in the weekly report, the number of defects decreased from week to week. Things like this need to be straightened out together because if it keeps happening there will be some bad things that might happen, such as:

1. Careless product delivery without deeper analysis on the quality side.
2. Lack of innovation in exploring and finding extraordinary edge cases or more complex possibility flows.
3. Worried that if further exploratory tests are carried out, more defects will be found so that it can worsen the existing metrics (the number of defects will spike much higher than the previous week).

Therefore, it is essential for us not only to be aware of the number of defects that appear, but it is also necessary to look at the details of the defects that occur. Is it a minor defect, for example, once you click a button, an error directly appears. Or is it a defect that appears after we trace various complex flows, such as:

1. Includes scenarios using back-and-forward action.
2. There is action to move to another app and come back.
3. Passing a path that people may not be aware of.
4. Includes scenarios using various network speed conditions.
5. Includes scenarios using various device types and screen sizes.

Speaking of defects, in the O2O tribe itself in Q2 2022, there was one key result as follows:

Reduce the number of bugs or defects per squad by 50%.

The Quality Manager and Head of Quality at O2O start to think about how we can achieve that key result. Long story short there are several approaches that we try to do, such as:

1. Reaffirm to every member of the squad that quality is everyone’s responsibility. The main focus is how we are able to influence, engage, and empower our team to have a better process, deliveries, and confidence without having to rely too much on a test engineer or quality coach.
2. Agreed on shift left testing and testing pyramid concept. We try to find another way to move the test closer to the engineers (shifting left), so they can see whether their code works or not by testing on lower-level tests (unit test, component/fragment test, API test). It will reduce time consumption compared to that if we have covered every test on a UI integration. In the end, we can also improve our mindset about quality.
3. Empowering and influencing more engineers to be more disciplined in doing self-test. With the engineer getting used to doing self-tests, when he develops a feature he will also think about various possible cases that can occur. They will also get used to thinking about the testability of the code they create. When a self-test is carried out and it turns out that a defect appears, they will be motivated to immediately solve the defect. So, when the self-test by the engineer is completed and continued with the exploratory test by the test engineer or quality coach, it is hoped that very few minor defects will occur.
4. Focusing on lessons learned from every defect that occurs. Empowering the test engineer or quality coaches to consistently discuss with squad members the defects that occurred and plan to prevent similar defects from occurring in the future.

So in conclusion, the right mindset about defects is not enough just motivation for pursuing zero defects, but rather how we have made efforts to find and prevent high-quality defects, how we regularly conduct defect analysis to see the details of defects that arise and gather the lessons learned. As well as not being satisfied quickly and always being curious about what other cases might cause defects.