Why I’m suspicious of A/B tests
They’re missing an element.
The other day I stumbled across one of those customer experiences that makes you want to tear your hair out—and preferrably, stuff it down the offender’s throat.
I recently started writing more and decided that I would take some time off from the side work I do hourly to support my Fittr/startup lifestyle. So I went to my freelance hour-tracking software service and looked for a “pause” option, so that I wouldn’t have to pay $20/month for 2–3 months.
These were the three options given:
And there, in the center, was the option I definitely wanted: to “pause” my account.
So I was excited it would be so simple, and clicked the button.
But I got an error message.
“Reduce the number of clients to 1"
They wanted me to go through and delete all my accounts? I couldn’t do that! I had somewhere around ten customers listed—and two of which were the only places I had stored their company’s contact information.
I looked at the choices again. Was it worth paying $20 to keep that information? Probably, yes.
And I wouldn’t have canceled at all—except that I was pissed. What a poor customer experience! And with righteous indignation, I copied a little information from the account and closed it entirely.
(but not before sending them a nastygram about their UX)
A few days later they wrote back to me and very kindly admitted that their cancelation policy was dumb. In fact, they even said their old policy was so flawed that—
“We made it easier for customers to cancel than to stay!”
So I felt a lot better. And when I start billing clients again, I’m going back. Their service really is excellent, and I had been very happy with it. I felt guilty for sending them a note at all, but their UX had been completely illogical, so maybe, I thought, my anger and nastygram had helped in a way?
I couldn’t stop thinking about it, though. How had they decided upon a user experience that would enrage a small group of customers?
And then I remembered that thought I’d briefly had: ‘maybe it was worth $20 just to keep the information.’
In an A/B test, I have no doubt that this would have decreased the number of cancellations.
After all, they had increased the pain of leaving.
What an A/B test doesn’t show, though, is how much a customer hates you for it.
…it misses the human element.
I mean, that’s not the only thing flawed about A/B tests. Don’t get me wrong.
There’s also the problem of statistical signficance.
If you search signficance of A/B testing, Wikipedia gives you the following scenario:
Email test group A: 50 out of 1000 (5%) success rate
Email test group B: 30 out of 1000 (3%) success rate
We’re all going to go with test group A, right?
Kiss Metrics provides this incredibly misleading tool, which gives the following ‘analysis:’
But then, when you play around with the calculator a bit, you realize something’s wrong…
Wait—800 fewer tests, and I still get the same results?
It doesn’t take the size of the pool into account WHATSOEVER. Idiotic.
(see sidebar for additional discussion→)
Whereas Optimizely provides this actually useful tool:
Basically, to test for significance of a 25% change off of an expected 4% conversion rate… well, you’d need to send it to more than twice as many subscribers than you actually have.
So yeah. Don’t always trust an A/B test without putting some serious thought into it.
My best suggestion? Narrow it down to a few options, sleep on it, and then in the morning pretend you are a new customer seeing your emails for the first time. How do they make you feel?
Empathy will get you a long way in UX.