So You’ve Got a Big Idea…

Mike Atherton
May 30, 2015 · 7 min read

Imagine you’re working for a large UK mobile provider as part of a team dedicated to improving the delivery of products purchased online.

Someone in the team has the idea that offering same-day delivery to people wanting the latest technology will increase purchases.

Maybe they’re right. Let’s find out.

State your assumptions

First let’s look at the idea as stated: ‘same day delivery to people wanting the latest technology will increase purchases’.

And restate it as a complete assumption:

“We believe that offering same-day delivery
For new purchasers of current-gen phones (let’s say)
Will significantly increase purchase conversion
We will know we are right when we see the rate of conversion increase by 20%, directly compared to not offering same-day delivery”.

So now we have a specific hypothesis, and a measurable goal. (That 20% figure is arbitrary for this exercise; in reality the target would be based on the projected cost of offering the service).

Testing the hypothesis

To test this hypothesis, we need to build a minimum viable product. In the first instance, we just want to assess whether there’s any demand for a same-day delivery option.

Our MVP therefore could simply be the presence of a ‘dummy’ same-day option during checkout. On selecting this option, customers are told ‘this isn’t available in your area yet’ and asked to select another option.

Aside: Anecdotal evidence suggests that this approach to Lean MVPs is becoming so prevalent that users are fatigued by placebo features. A further hypothesis could test whether this is so, and how users may be positively incentivised.

Knowing the audience

Proto-personas (and validated personas) summarise research findings in customer segmentation.

Who are these ‘people wanting the latest technology’? We’ve been doing this a while, so we’ve probably already distilled our ongoing user research into behavioural persona types. Circulating the ‘gadget freak’ persona to the team will remind them of relevant customer needs, wants, and pain points, and serve as stimulus for the design process.

Designing the MVP

While potentially simple, the cross-functional team should collaborate on options of how to incorporate the same-day feature. This includes its place in the user journey, the design and positioning of the call-to-action, interaction patterns for selection, and options around language to promote trust and understanding.

Everyone designs. This could be done through freeform collaborative sketching or a structured design studio exercise, rapidly exploring and iterating different design concepts before converging on a design hypothesis.

Collaborative sketching sessions yield many design variants quickly and cheaply.

If deemed strong enough, this proposed solution could then be implemented ready for running the MVP experiment. Often though, it’s a good idea to run some usability testing sessions with target customers.

Because awesome.

These can be done with low-to-medium fidelity prototypes — even prototypes created from the original sketches. The point here is to correct any usability issues (including whether the call-to-action is visible and understandable), so that we can focus our MVP experiment on desirability.

Running the experiment

We want to test whether offering same-day shipping is desirable to our customers when buying current-gen phones. So for this experiment, let’s say that if the customer has selected an iPhone 6 or a Galaxy S6, we’ll put up our (fake — shhh!) same-day selector among the delivery options in the checkout. When the customer selects this option they’ll be told that same-day shipping isn’t yet available in their area. Maybe we’ll also give them the option to be notified when it is. (If the customer is prepared to pony up their email address, we’ll know we’ve really got their interest). And if we’re feeling super-nice, perhaps we’ll give them discounted next-day shipping as compensation.

We’ll aggressively track how often the same-day option is chosen, which will give us an indication of desirability. We’ll also track what’s selected most often after the customer realises there is no same-day. And cart abandonment after same-day selection, compared to abandonment with other shipping options. How many people want their phone today or not at all?

Tools like Google Analytics are great for measuring the outcomes of the MVP experiment.

Of course it could turn out that hardly anyone chooses same-day after all. We were careful enough to rule out usability issues during prototyping, so we’re pretty sure it’s actually the idea that’s a dud. Oh well, no harm done. At least we didn’t go to the expense of building out a same-day logistics infrastructure.

But let’s stay positive, and agree that our little dummy button is getting a lot of love. On we go.

The ‘Wizard of Oz’ MVP

We’re confident now that there’s enough interest in same-day shipping to keep going.

Returning to our stated assumption, what we actually said was that we reckon offering same-day delivery on the newest phones will increase purchase conversion. In other words, not only do people want same-day, it’ll be the deciding factor in closing the deal. To test this we might actually have to offer some same-day shipping. Yikes.

But before we start buying up fleets of trucks, remember this is only an experiment. It can be small, and our solution doesn’t have to scale.

Aside: When Nick Swinmurn started Zappos he wanted to be sure there was a market for buying shoes online, before he bought inventory or built out a fulfilment and logistics chain. So when a customer placed an order, Nick would drive to his local shoe store, buy that pair himself, then ship them to the customer. Based on successful validation of the business model, Zappos then sought funding for scaling up.

Assuming our iPhone 6 and Galaxy S6 are selling pretty well anyway, it’s only fair if we can directly compare two parallel worlds where the only difference is the offer of same-day shipping.

Enter A/B testing. Based on IP range or user accounts, we can fix it so only certain customers are offered same-day shipping.

Since we’re experimenting, we’ll be providing shipping by some manual means. That probably limits us to only offering it within one or more defined regions. Reviewing our customer research, we can figure out areas where sales of current-gen phones are steady enough to collect sufficient data for comparison.

Cut to the chase: A customer in Manchester orders a Galaxy S6. Striking from a hidden base nearby, our rebel experimenters take a phone from their allocated stock and send it out on a taxi or an Uber, getting it to the customer within a couple of hours. Rinse and repeat this procedure for all iPhone/Galaxy purchases in Manchester for the next two weeks.

Measurement

Every customer who gets this treatment also gets a follow-up. Maybe a survey, maybe even a phone interview. Was same-day a major factor in their purchase decision? Were they happy with the service they received? How likely would they be to use it again? Would they recommend it to a friend? We want to know.

Meanwhile, we’ve also had a keen eye on phones sold in Birmingham — determined to be similar enough to Manchester’s sales profile to serve as a control. Remember — customers in Birmingham were not offered same-day shipping. Not even the fake button.

The big question: when normalised for population scale and other factors, did Manchester sell at least 20% more phones than Birmingham over that two-week period?

(Of course, substitute more relevant areas here, according to data and practicality.)

Hopefully the numbers alone will give a clear enough signal of success. If in doubt, we can compare it to the qualitative feedback to get an overall picture of how we did.

Extra credit: Testing pricing

As part of either our first or second experiments (or even as a third) we probably also want to test how the price of same-day shipping affects conversion. Using a multivariate setup on a large enough sample, each eligible customer will see one of three different prices. Behind the scenes we’ll be tracking conversion and projected revenue.

Go big or go home

Mission accomplished. Hypothesis validated. High-fives and Hob-nobs all round. People want same-day shipping, and more of them actually buy when it’s offered. Now we just have to do it for real.

Eventually same-day might be truly part of our own supply chain, but for the medium term let’s take a ‘small pieces loosely joined’ philosophy and partner with a same-day service.

Shutl (recently acquired by eBay) are exactly this. They offer pluggable same-day fulfilment, with an integration method analogous to PayPal. In other words, you can offer Shutl as a delivery option on your site which they hope will become a readily-understood and trusted shorthand for same-day delivery.

Aside: Shutl’s own UK customer research has uncovered skepticism that same-day (sometimes same-hour) fulfilment really works. They’re working on education and building trust.

Even as we scale up we continue to track conversion, test pricing, and measure satisfaction. We’re always in a cycle of building, measuring, and learning.

The experimentation never ends.

Mike Atherton

Written by

Creator of content, builder of brands, architect of experiences. Available for children’s parties.

More From Medium

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade