Marginal Utility and Taco Consumption

A Tasty Analogy for Regression Testing Suite Size

“Marginal Utility to Variable Inputs” That sounds like the mother of all exciting conference breakout sessions, doesn’t it?

Well, it may, but it’s also an important economic concept. People probably practice it, even if they don’t know what it’s called. In this post, we will look at this concept as it relates to optimum regression test suite size. In order to do that, we need to define a few working terms.

The first of these we will call ‘utility’

Economists refer to it as the useful-ness that a consumer gets from a good. It isn’t measured always in money. Utility can be measured in the security you feel from having anti-lock brakes or in how much you enjoy your favorite television show. This is important to the ‘marginal’ part of the title statement.

The second concept is ‘marginal’

This refers to the last unit of whatever you are measuring. So, if you combine ‘marginal’ with ‘utility’, we are talking about how much you will enjoy the next thing you are considering buying/using/doing/watching on TV/etc.

Being able to measure that marginal utility is key in determining when to stop doing something.

Two Taco or Not Two Taco

Let’s take the case of late-night, college soft taco cravings. Hearken back to the days of being 21-years old — you ate your last D-hall food at 6:30PM, it’s now 1:30AM, and you are craving some meat, cheese, and a soft tortilla. It’s entered your lizard brain and you can’t get out! So you head to the local late night Mexican restaurant with the intention of ruining your previous best total for soft taco consumption. Only this time, you will do it without the secret Habanero sauce (the green stuff you have to know about to ask about, sort of like speakeasy protection on servers). Let’s just say, you learned that lesson the hard way.

You eat your first taco, nay, inhale it, and think, “Is this just a small slice of Nirvana, here on earth?” Well, let’s measure that. Let’s say that your total feeling of awesomeness (in this case, a substitute word for ‘utility’) goes from zero to fifteen units on eating that first taco. So, what was your Marginal Utility?

Fifteen.

This is also your Total Utility. That is, until you eat your second taco; maybe a hair slower than the first, pausing to chew and savor that amazing flavor.

Now that you have consumed your second taco, it’s time to re-contemplate your Total Utility (Awesomeness). You are definitely better off than before you had any tacos, but are you twice as well off? The Laws of Economics pretty much say you are not. Let’s say that your Total Awesomeness is twenty-seven now. So, what was your Marginal Utility?

Twelve.

That means that, for this marginal unit of taco, you had a decreasing marginal return of three units. It was still a positive return, but not as much as on the first taco.

This concept should be starting to take shape now. So let’s eat one more taco and then look at a chart.

Third Taco’s A Charm

Taco number three. It is still delicious, still good, and still makes you wonder if this is proof of divine preference for your soul. But now you are starting to notice that it isn’t quite as good as the first two. Maybe those were a little warmer and fresher. Maybe this isn’t the highest quality lettuce in the world. That said, your Total Awesomeness has reached thirty-seven. This is a marginal return of ten; which is two less than your second taco and five less than your first.

To return those numbers to normal-speak, “You are enjoying your tacos, but each additional one is enjoyed a little less than the last one, even as your total enjoyment level is going up.”

As promised, a chart:

How Does This Apply to Regression Testing?

As your late night taco binge continues, you will notice a couple of things. Eating your fifth, sixth, and seventh tacos does not have as high a marginal return as your first through fourth. In fact, after, eight tacos, you begin to show a negative marginal return. This makes sense; after nine tacos, anyone would start to feel a little bit sick and question their place in the world.

This all begs the question: “How does this explain regression test suite size, especially as applied to upcoming CI/CD processes?” Well, we have a finite resource (time) in which we need to fit all of our regression testing so we need to make certain that we get the best balance of total return and marginal return from our variable input, the test suite.

What are the high-value flows for a website? The flows that need to work every time? They’re probably flows like Login, Forgot Password, Change Username, etc. These are our first taco and we need to test them first EVERY time. So these will definitely go first in the test suite. Do we put habanero sauce on them? (do we do all possible logins from all entry points from all types of users) Probably not for this taco, but for later tacos, for sure.

Our second taco should be the security services that ensure the safety of customer data. These also happen to be easy to test, as they are usually APIs or straight-forward tests. We probably would not consider too many non-customer facing tests at this point.

What is our third taco? Anything new from our last three releases. Once that becomes old hat, maybe we can cull them out of our regression suite, but for the moment they stay in. Our fourth taco? Maybe some tests that failed in the last four or five cycles. Our ninth or tenth or eleventh tacos? This is where we get into extreme edge cases.

Regression Testing in Action

Not everyone connects with abstract analogies. Then it becomes important to use dollars as the unit of measurement.

When measuring costs, some items, such as the hourly cost of a tester, are explicit. Other items, such as customer goodwill and brand trust, are less easily measured. For the following use cases, we’ll look at simple explicit test costs using taco customers as an example. Please keep in mind that the figures used are fictional examples used for the sake of this post only.

Picture Steve the Taco Delivery Guy. Steve lets people order their tacos through his website and he delivers them to their door. Customers have their own usernames and passwords and their account stores credit card information and favorite orders. When a customer can’t login to access their orders, they have to call Steve to place orders directly. Calling Steve costs him the time it takes to create three orders whereas using the website means Steve can fill orders at full capacity.

For each hour that customers cannot login, our taco entrepreneur effectively loses $50 per hour. If his normal orders total $250 per hour, when customers cannot login, he is losing 20% of his business. Furthermore, customers may become annoyed or nervous if there are continuous problems with a website. Therefore, continued failed login attempts means that Steve is losing brand value as well.

Steve will need to make certain that his first, and most frequent, testing scenarios involve his customers being able to login. These are his first tacos.

Let’s look at other scenarios to see where Steve is best served testing his website.

In Scenario #1, Steve’s customer has forgotten their password. This is not a problem normally, as customers can self-serve to recover a password. However, when the self-service software fails to work correctly, Steve has to take calls. These calls cost him about 10% of his business, but only when the software fails.

If we were to use our examples of testing as tacos, this password recover scenario would probably slot in around the second or third taco.

In Scenario #2, a customer lives too far away for delivery. Steve the Taco Delivery Guy has expanded his business to cover the entire City. However, on occasion, he gets an order from outside the delivery area. Usually the system detects this before the order is finalized and the account charged, but if the user has an especially old browser it does not. The result is that the order gets made, the driver realizes it is too far to deliver, and the order cannot be fulfilled. This happens four or five times a month; each time, this means a loss of the food cost and potentially some customer goodwill.

When Steve inquires with his website contractor about detecting these kinds of potential losses, he is told that the cost to test these scenarios is pretty high relative to the losses incurred, but it could be done each time a major browser has a release. This necessarily slots these test cases down around the far end — our fifth or sixth taco — since they are expensive to run and have very low marginal return.

Beyond Tacos

Our two scenarios represent two simplified ends of a spectrum, but it should be clear that from a financial standpoint, any regression suite for our fictional Taco Delivery Guy should have more customer login use cases than geographic/browser detection cases. This is the best way to balance total return and marginal return and get the most from our tacos.

So what can we learn about testing from all this taco eating? Make your regression test suite flexible and designed with marginal value in mind. Keep it risk-based. Keep it nimble. Your team will save time, money, and frustration.

Also, no one likes to get sick off too many tacos.

Originally published at developer.capitalone.com.

For more on APIs, open source, community events, and developer culture at Capital One, visit DevExchange, our one-stop developer portal. https://developer.capitalone.com/