Analytic Lessons in Experimental Design
Inspired by Popular Hip Hop Songs
Analytics is a discipline unto itself. And yet, since not long after Aristotle, it has been a multi-disciplinary study. Music is fun. In the form of mnemonics, it can be an important and serious component of learning.
Experimental design is a difficult and serious component of both analytics and the scientific method. This article, while addressing a serious subject matter, is light-hearted and fun. Enjoy the read and click on the bold links to enjoy the music.
Great Experimental Design is Not Easy
Analysts should never underestimate the difficulty of constructing good experimental design. This issue was best stated by a famous team of data scientists from Hollis, Queens.
This blog is my recital,
I think it’s very vital
To build (a test), designed (the best)
Experimental Design’s the title,
Here we go.
It’s tricky to build experimental design,
Experimental Design that’s well-designed.
It’s tricky.
The following sections will outline important steps to consider before you Run an experiment.
Consider Removing Outliers
Well-designed tests should have populations that are normally distributed though a handful of other distributions are also easily assessed. In most cases, you will want to remove outliers from your population. These residents of the long tail will obscure your findings and ruin your averages. In the words of yet another data modeler out of Queens:
I’m gonna pull you out and let you understand
That your not an average man
When I got criteria in my hand
Damn!
Oh!
Listen to just what I say
I’m gonna pull you out
Outliers just have to come out
I’m gonna pull you out
Outliers just have to come out
Remove these outliers before you begin your test. Stay Cool and rely on simple criteria rather than over-engineering your population.
Measurement Requires both a Test & Control
Designing a good experiment requires that you set up a minimum of two matched populations. Aside from screening outliers, these populations should be randomly selected from within your target populations. Never be tempted to deal with a control after the fact, while not impossible, it is poor form.
Once you have your populations divided choose one to be your control. Remember the words of those master marketing analysts from Harlem:
It takes two to make a test go right
It takes two to keep your findings tight
Split it!
The situation that the test is in
I’m looking for facts so I don’t wanna bend
Experimental design for a good end
But listen up ’cause I want you to comprehend
’Cause I’m the leader, the man superior
If you don’t make two your test is inferior
When it comes time to measure your results, you will Base all of your lifts and impacts by comparing one to the other.
Don’t Compromise Your Control
Once your control has been separated, do not let anything compromise your test. Your control should not be influenced by other tests, unusual treatments, or any other compromising influence that you can prevent. Remember the warnings of that senior data scientist from Oakland, California.
Give me a group or control
Keeping ’em clean that’s what I’ve been told
Now they know when you talk about the testing
You talk about a control that’s held and tight
Marketers are pressin’ don’t buy the hype
Or make a change to learn it’s a mistake
And now we just can’t learn
The charts you get won’t reward
So you might as well quit
That’s word because you know
Can’t touch it (oh-oh oh oh-oh-oh)
Can’t touch it (oh-oh oh oh-oh-oh)
Break it down
It is important to Hammer home that a control often requires its own treatment, if keeping it untouched would not be representative of ‘normal’ activity.
There is Nothing Wrong With Sampling
If your populations are overly noisy or your marketing team is overly sensitive to maintaining too large of a control group, remember the wisdom of the marketing master from Miami.
If there was a problem, sample to resolve it
Use random digits as an easy way to solve it
Sampling can save you when you are Under Pressure. Don’t get carried away. Any off the shelf sampling product of random digit generator will work. Even if it feels Vanilla. Do it right and you should have a great story for your marketing clients.
If You Can’t Repeat It, It Isn’t Real
You thought through your design. You removed the outliers. You create a test and control. You kept the test clean from unwanted influences. Maybe you even utilized sampling. And when you measured the results, you want to shout with joy. But can you repeat it?
Upside down and inside out
I’m about to show all you folks
What’s it’s all about
Now it’s time for an analyst to get on the mic
And test this hy-pothesis right
I’m taking it back to the old school
’Cause I’m an old fool who’s so cool
If you want to get it right
I’m gonna show you the way, gotta do it twice
‘Fore you have something to say
When that data first comes in with double-digit lifts, you will be tempted to give a little Whoot-Whoot. But only if you can do it again, say a month later, can you really say Whoomp, there it is. Incidentally, I believe that is four words?!
Risk… Don’t Pretend
Take risks when you design your experiments and never be afraid to fail. If you do, just don’t try to pretend it actually worked. There is nothing wrong with an experiment that found nothing. That is learning in itself.
He’s going to have a hard time explaining to management
These bad designs and weak tests
Cause he never risked it, he hoped and he wished it
But it didn’t fall in his lap so he ain’t even learn it, he pretends that
Can we pretend that weak tests
And bad designs
Are like good results
I could really use a pass right now (Pass right now, pass right now)
Take smart chances in the name of learning. Even if you fail, there should always be a part 2.
Think about ROI
Our last bit of wisdom comes from the Lion of experimental design.
Rollin out my test, gotta control, measuring ROI
Laid back [with my mind on my money and my money on my mind]
Rollin out my test, gotta control, measuring ROI
Laid back [with my mind on my money and my money on my mind]
It’s a dog eat Dogg world. If your test does not figure profitability into the results, you are testing for an inferior outcome. Be sure to measure ROI, even if it means you need to wait a while longer for that profitability to become apparent. Many a business was ruined optimizing on open rates, sign ups, and views. They should have been measuring purchases, subscriptions, and orders.
Follow these steps and you’ll be singing hip-hop hurray!
Thanks for reading! If you enjoyed this, please click the recommend icon ♥.
Article originally published on LinkedIN.
