When All Other Testing Fails — Bootstrap
An Analyst’s Last Resort For Harvesting Insight
In this series, we have covered the often over rated A/B test and the often overly dismissed PRE/POST. If you recognize that A/B testing basically includes Test/Control & Champion/Challenger and that PRE/POST can itself have a few other names — you should realize we have covered most of the testing landscape already. But where do you go when all else fails?
Let me stop there. What does “all else fails” mean in this scenario? The answer is not a scientific one. The scientific answer is — when all else fails — do a proper test and control. If that fails… give up.
The business answer is far more practical and complex. What if you just can’t do a proper test/control or pre/post? What if there is no time, no money, other considerations? AND what if there is a LOT on the line — so giving up isn’t going to work?
NOTE — fails = “fail to test” NOT “fail to get the answer you wanted”. The latter is not a fail … it is a personal disappointment.
So if you have a decision, if no other method can help, and if you can perform it — you might just try the A … I mean bootstrapping a control. Got it?
Actually — what the hell does bootstrapping a control mean? The answer — put simply — is creating a look-a-like population, typically after-the-fact. If you do it ahead of the game, it is just called sampling. Sampling is more reliable for that very reason. Because you do it in advance — it avoids a lot of nasty bias. Of course sampling is just A/B testing… and we know that has already been eliminated.
So how does one do this? The answer is simply and logically — and with a massive amount of skepticism. Let me provide an example.
So let’s create an unlikely scenario — but one that hopefully sounds like an unsolvable issue an analyst might need to confront. A social media company created a campaign featuring Logic the Rapper. Perhaps it was to fulfill a contractual obligation with some other client — yes, academics that happens in business. For some reason there is a belief that this particular approach doubled activity among Fortnite players. (I would hope activity = $ … but I am staying as vague as possible). Finally — there is a window of opportunity to secure Logic at fractional prices for a second, targeted campaign — but the decision needs to be made in 48 hours. Hey, the man is busy!
I would assume, in this example, we have some means of targeting Fortnite players. I also assume that in looking at the earlier campaign, we clearly see a doubling of activity. The problem then is why? How do you know the campaign caused the activity… maybe the Fortnite servers crashed at the same time and there was literally nothing else to do (ok, that is only literal for fortnite players…). Finally, I would assume pre/post is not possible due to limited time series or problematic seasonality. If pre/post was discarded for severely low volume or intense noise — bootstrap at great peril!
So to bootstrap a control here, we would try to use the campaign criteria on a new population. I would then advise comparing the new population on a few other dimensions — first and foremost — the percentage of Fortnite players. If they do not compare favorably, you need to know why — before you try anything else.
Assuming the bootstrap population passes that test, I would check a couple other standard dimensions — age of the customer, age of the account, maybe a state or geographic split. Remember — you WANT a look-a-like control. Also remember — if you have to go to great lengths to force a match… well, I wouldn’t. If this doesn’t prove fairly simple and logical, I would actually bail.
BUT assuming it really was pretty simple, logical, and did look-just-like — congratulations you have a control that should allow you to harvest real insights! Feel free to dance. And feel free to dare… catch that because that is what you just did. You built a control group that should give you more piece of mind that a course of action is likely to result in the outcome you want. YOU HAVEN’T PROVEN ANYTHING. That requires — as always — that you can repeat the results AND that they actually move the P&L.
Thanks for reading. We still have a few more insights to share in this series centered around the process and other external factors. So, stay tuned.
Want to learn more about real-world testing issues? Consider -
Don’t let your analysts bootstrap without it!
