How Hypothesis Testing Killed the “Best Practise”

Free your mind — challenge product “best practices”

  • We (*) altered the registration process, so that half of the new registrants would receive life-cycle emails after their registration.
  • The other half, would not receive any (null hypothesis).
  • The system would inject a Google Analytics (GA) custom variable in both cases, so we could later on segment any reports and see differences in usage, retention, etc.

The result

Note that Group A ended up having slightly more users than Group B, which could account for the 1.4% difference.

So are “best practises” wrong?

Is X better than Y? Is practise Z a best practise? Well, “it depends”.

Stay skeptical, stay doubtful

  • Not take “best practices” for granted and test if they work for your audience.
  • Evaluate your alternatives with as much data as possible.
  • Your “gut feeling” is good. Data is better!
  • Be ruthless with the features you build; if they’re not of real value, put them out of their misery — fast.

Further reading



Helping dev teams succeed by adopting new technologies and practices - &&

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store