Designers — launching without testing is stupid

Stop doing it!

Every public facing design you push live — which has the potential to affect core metrics — should be A/B tested. Period. End of story.

Why is testing so important?

A couple of reasons:

  • Data helps you make informed decisions. You test to learn. You’ll find out what works, and what doesn’t. You can then share what you’ve learned with others, and apply what you’ve learned to future designs.
  • As much as you’d like to think that you can predict success, humans are terrible at it. There are no exceptions to this statement. That’s not to say that you don’t have a wealth of knowledge and experience that you can leverage. You’re always going to be fairly confident that each design you launch will lead to an increase in your core metrics (else why would you be launching the design in the first place). But just understand up front that half of the designs you release are going to be a stalemate or a bust. That’s just the nature of the game, and if you’re not testing, you won’t know which half is which.

While testing may seem burdensome, I promise that as you start doing it, it will get easier. Eventually, you’ll be able to launch tests in a fraction of the time it takes you when you first start out. You’ll begin to see with absolute certainty what’s working, and what isn’t. As a bonus, you’ll be able to prove exactly how much money you’re designs are making for your company (say hello to a nice little bump in your salary this year…). It can actually become very addicting, and can be very fun.

Tests, I don’t need no stinking tests, I’m already a great designer!

Again, by all means, please leverage your instinct and your experience as a designer to initially come up with the best design possible. But then please test each design, as a safety mechanism to make sure your assumptions were correct.

If the test succeeds, celebrate! Great job.

If the test fails, celebrate! You just learned something new about what doesn’t work, and you avoided launching it to your users! Now share what you learned and use your new found knowledge to keep iterating.

“We launch without testing, but we keep an eye on our stats”

Let’s call this what it is… You’re being lazy, or you’re letting your ego get in the way of good design.

Launching something and watching the numbers may (potentially) work — perhaps if you only launch one thing every two weeks...

But it doesn’t scale. As your number of commits in a company increases, this “just launch and watch the numbers” approach will fail you.

Think about it, let’s say you launch 6 new features in a month, and as a result, a month later you start to see a slump in your core metrics, which of those 6 features do you attribute the slump to? Or is it something else completely?

Since you didn’t test any of the 6 features individually, you’re completely in the dark as to what’s causing the slump, and you’ll likely waste a weeks worth of additional time trying to figure out exactly what did cause it.

Conversely, If you did take the time to test each individual feature, you’ll know exactly which features performed well, and which didn’t. With each feature you’ll have the insight you need to make informed decisions:

  • Do we go ahead and launch it anyway?
  • Do we just scrap it?
  • Do we tweak the feature a bit to see if that makes any difference?

The important bits

Eliminate ego — In order for any of this to work, you’ve got to drop your ego. You’re going to instinctively want to mask every test you run as winning in some way (you’ll always be able to find some number that goes up and right). Don’t do it. Just know that 50% of your designs are going to be stalemates or flops, and a good portion of your wins won’t actually move the needle that much.

Get in the habit of testing everything — It’s best to get in the habit of testing every design you launch. When you launch designs without measuring them, you learn nothing about which designs work and which don’t. That’s a shame, and a waste of your time/resources.

Focus on learning — Whether a test is a success or a failure is unimportant. The important part is that you learn something new with every design you ship, and that you keep on iterating.

</rant>

Show your support

Clapping shows how much you appreciated Dave Martin’s story.