Confident Testing: A Methodology of Testing at Scale

I love teaching people about Continuous Confidence. Testing Software can be difficult and sometimes quite frustrating. It can be even worse when you have an established suite of tests that keep growing and growing over the many years.

Testing, however, is far more than “kicking the tires” or “verifying it works.” Testing can be an incredible experience, that when harnessed, makes a better Developer and separates us from a mediocre average developer to an elite developer. The difference, however, is not in whether Unit Testing or Integration Testing or End To End testing is right or wrong, it is how we use these tools. What we are trying to achieve.

Why Do We Test?

So, we need to ask ourselves before any test we write: Why am I testing?

Is it for coverage?

I think it hilarious when people approach me and start talking about their test suite which has a really high near 100% code coverage. They are so proud of the fact that they are there. They exude pride in their accomplishment.

I admit, it is quite the feet to have 100% code coverage. The next question I have is: How are your bugs? They begin to tell me they are about average as everyone else. That being true, then if they are a medium sized shop, releasing 100,000 lines of new code a month, they are releasing around 666 new bugs per month in a Software As a Service(SaaS) model.

The automation suite is covered. They have QA run through their regression suite every month. It takes 1–2 weeks, for maybe 20–30 QA people. Back and forth making sure we have tested it all.

So what is important, why are they testing? To cover code?

Is it to not break Production?

Others come to me to discuss the way they release code. They have a large End-To-End Regression suite, often resulting in switching to a Manual Test strategy, resulting in rather large release windows and releasing large changes of code at once.

You know the situation, the scenario: We are Enterprise software, enterprise software needs to be more stable, so we will only release 3 times a year. So, come release time, it’s all hands on deck! They have an automation suite that runs, rather large, and quite lengthy. Once that is complete, they take the time to make sure that QA runs through their regression suite.

Now, remember, they only release 3 times a year. With a medium team, they are releasing around 400,000 lines of new code to production with each release. They are trying to not break production. They are making sure process if followed. Yet, when I ask them how are the bugs, they say: about average. So that means around: 2,664 bugs for each release, around 666 bugs per 100,000 lines of code.

So, what is important, why are they testing? To not break production?

A New Reason to Test: For Confidence

In reality, the underlining reason why all people are writing tests, is to gain confidence in what is releasing. However, something that is not truly apparent, and causes more hardship than anything else is the methodology of testing. The core of the methodology we use in testing will determine what are are focussed on and how effective we will be in our test suite.

Let me provide some examples of what I mean by our methodology drives our focus.

Example 1: Code Coverage

Back to our initial example. When we focus on getting code coverage, we often begin focussing on simple getting things covered. We write tests that are often superfluous and in-effective. Let’s take a java example:

public class Testing {
    private String foobar;
    public void setFooBar(String baz) {
        this.foobar = baz;
    }
    public String getFooBar() {
        return this.foobar;
    }
}

When talking about this simple class Testing, we obviously need to achieve 100% code coverage so we need to write tests to set and get the foobar variable.

Let me ask you something? Do you trust the Java Language? If I couldn’t set a variable in a class, inside of java, then I have something FAR more problematic than my code not working. Nothing would really work nor compile.

Writing tests just to have code coverage near 100% results in superfluous tests and bloating our suite. When we see a large number of tests, we believe we are covered. This leads to False Confidence in our system. It causes problematic releases and more bugs than we should have.

Example 2: The Large Regression Suite

You have seen it. You know what I mean. A regression suite with 10’s of thousands of test cases that we need to run through in order pass off a release is good. The kind that sucks up your QA resources. When we take a lot of time to release code, we often disregard the testing phase until the end.

We know what needs to be tested, but we push on. Things that don’t meet our deadline, or tests that we didn’t have time to automate, are pushed into the Regression Suite to make sure they are handled manually.

Over time, that manual test suite becomes large and very time consuming. This results in our pulling QA resources in and making sure that each test is done. As the suite becomes larger and larger, we take time from our normal development time, and we start to automate the Regression Suite.

Here in lies a problem. If we are automating the Regression Suite, we are testing things that are not new. We are automating the old code while blindly releasing the new! As we pull more and more people in, we are actually creating a situation where we don’t know what happens when there are problems.

Generally speaking, when have this kind of methodology, we are testing via Integrations and End-to-End tests resulting more than normal time to release code.

Think about the implications of this methodology. We test more towards the end of the cycle. We implement slower more resource intensive test methods. If we release once a month, and we require all of a QA organization to regress that release for 2 weeks, then we are saying every 6 weeks, we run the regression suite. This means, 17 weeks of the year or 33.33% of the QA budget is dedicated to a manual regression suite. If we spend $1 million a year on QA that means $333,333 is spent on just our Regression Suite and nothing else.

The Regression Suite becomes a black hole for QA resources and time. But, when the suite is large enough, it becomes also, redundant. In my experience, when we have an automation suite and a manual regression suite, approximately 50%–60% of the regression suite is overlap to the automated suite. That means, more than $150,000 is simply wasted with no return on investment.

Example 3: Confident Testing

When we focus on being confident, our methodology drives a different result. When we test, we want to test to make sure that we meet the spec. But we test it in a different way. We follow the pattern:

1. Test what you change
2. Test your Logic
3. Test your Contracts
4. Test your Assumptions

For Further information see: https://medium.com/@continuousconfidence/confident-testing-quality-automation-testing-197be5180899

This methodology forces us to address the changes to the system we are making, not just the results of the system after the changes are made. What I mean by that, is Confidence comes from White Box testing mostly rather than Black Box testing.

Instead of focussing on the release of the code, we focus on the merging of the code. It causes us to test for the Change Request, prior to merging into the code base. It forces us to make smaller changes. Meaningful changes. It calls for us to slow down and understand how and why we are making those changes.

The incredible part is that the VAST majority of the bugs created in code actually derive from errors Logic, Contracts, and Assumptions.

This methodology will cause us to focus more on small meaningful Unit Testing rather than superfluous coverage. It will result in a test suite that takes minutes to run rather than hours to days. It will also, more than likely result in the absolution of the Regression Suite we feel we need today.

All of this will allow us to raise the Confidence of the Developer, the code quality, and the overall stability of the application as whole.

Still don’t believe me?

May I share some interesting statistics from a company I have witnessed this work with?

1. Releases code 10–14 times a day with 0 down time.
2. Averages between 100–200 thousand lines of code a month.
3. Often closes more bugs than opened in a given month
4. 1.9 bugs / KLOC — 1.9 bugs per 1000 lines of code (Industry standard is 10–15)
5. Automated test suite grows between 500–1500 tests per month
6. Automated test suite takes approximately 6 minutes to run. (More than 40,000 tests)
7. Efficiency to release new features went from 6 months to 6 weeks.
8. No Regression Suite at all.

Truly, when you focus on Confident Testing, you can do miracles in the Software world.