Escape the Groundhog Day of Testing with a Hypothesis Kit

“The newspapers of the twenty-first century will give a mere ‘stick’ in the back pages to accounts of crime or political controversies, but will headline on the front pages the proclamation of a new scientific hypothesis.”

Nikola Tesla

I like Tesla — he’s an awesomely cool dude. What he’s pointing out is the abuse of data to make a point that’s not true or is weak, lacking in evidence or frankly unsupported.

If you’re a fan of statistics, lies and the misuse of numbers in everyday life, you should listen to the BBC Programme, More or Less. Debunking myths, lies, illusions and numbers — and damned interesting as well as illuminating.

Whilst we’ve all probably read newspaper articles about this weeks spurious health claims, nowhere is the problem of numbers and leaps of faith so bad as on the web. Nowhere is the problem quite as evident as with the wonderful business of building digital experiences, products and services.

How many times have you heard someone say:

“I have x years experience” or

“I know what our customers want” or

“People don’t want it that way” or

“That won’t work” or

“This isn’t on brand” or

“This will work better for our customers” or

“I know what they want”

How the hell do they know, eh? What special anointment resulted in them being the supreme being — the one and only arbiter of what every customer feels, thinks and responds? Who put them up for the job of ‘The One and Only Customer Champion’?

Introduction

So what is this article about? Guessing! Yes — let’s get this one out in the open!

We’re all being told to use “hypotheses” in our work these days, so we should unpack what we mean here, how hypotheses are used and why it’s such a critical element in everything we do.

If you are doing any of these, then you are thinking of “If we do this, then X happens”:

Personalisation

Split testing

Targeting

Advertising

Product sprints

Launching a new app

Updating your product

Changing your page templates

Doing SEO changes

Putting in a responsive site

Changing the fonts

Optimising the checkout funnel

Can you see what I’m leading up to? Any freaking change to anything in your product — then you are forming a hypothesis. You wouldn’t just do this stuff not caring what happened, would you? At this point, I am laughing my ass off ironically, because this is what mainly happens.

Many times there is an expectation that something good will come of any change but the outcome isn’t always measured properly to see if it actually has done so. Analysis remains ‘post event’ — where any evidence might be siezed upon to prove or disprove what happened was positive — a post facto, confirmation bias riddled world of decision making.

Knowing what happened when you change any of this stuff is useful business knowledge. It’s a form of Intellectual Property that a business builds over time through decent experimentation and learning. A collective wisdom about where the profitable and productive areas are for discovering opportunity.

To me, that’s almost everything that happens with a web product — but hardly any of this is measured by companies, never mind it turning out to be productive in terms of learning. The cry of ‘We are too busy to measure what we’re actually shipping’ might seem illogical but it’s reality.

In this set of six short articles, I will outline:

(1) What is a hypothesis? Why is it important?

(2) Why agile testing doesn’t always result in velocity of learning

(3) Forming Hypotheses

(4) The Hypothesis Toolkit

(5) Deflecting Stupid or Needless work using a Hypothesis

(6) The Cultural Impact on testing

We’ll also introduce you to the Hypothesis Toolkit, which helps you express your testing ideas clearly and simply: