How to Improve Product Experiments

Tips on getting better at managing product experiments

Really understand your hypothesis

Know exactly what it is that you are testing. The hypothesis in a product experiment scenario would be “Solution X solves Problem Y”. Solution X would be your product.

The simplest solution is often the right one — Occams razor

I heard and learned this phrase over a decade ago from my English teacher, I didn't really appreciate it at the time but our teacher said it was very important and I should memorize it, so I did, later on I started understanding the importance of this simple quote and its validity for a lot of things in life. True genius comes in finding the simplest possible solution and nothing less.

Make sure you are certain that the solution you have identified is the simplest, once this is done convert it into software (i.e. your MVP)

Test within a 72 hour window

Once you have identified the simplest solution try to spend no longer than 3 days from the day you come up with the solution to the day you interact with your first customer or user. The general principle would be to create something that is “functional” — i.e a working prototype would be ideal before the 3rd day has ended, if not create something like an interactive mockup and find a user or customer to show it to.

Don’t be a perfectionist at this process.

Set your 1 key metric

Your experiments should be quick, simple and targeted. Therefore you need to gather data that can help you in validating them and hence you need data that is accurate, relevant and data that can help you make a decision on whether the solution you have come up with is the right one for solving the initial problem. So what I usually do is identify 1 key metric that I will focus on validating for each experiment, this gives me focus. Something as simple as user signs up, user completes form X or any other simplified but important call to action, but limited to just one metric.

Choose between Product or Customer rotation

If you are trying to validate product/market fit you can try testing with a few customers in a specific market, but later maybe you find that the specific solution just doesn't solve the problem the way the customer wants. At this stage you should decide whether you continue your next experiment with the same customer group by tweaking the product from what you have learned and seeing if the newly tweaked product satisfies the same customer group and then you can keep repeating that until product/market fit or you can try rotating the customer group with the same product.

For example, on a recent experiment at Gink Labs we did an experiment for a product we are working on called Placer and initially decided to target established ecommerce retailers but later decided to target agencies who were planning campaigns for lifestyle brands, the first customer group didn't quite turn out to be the right selection as they were not familiar with influencer marketing and were struggling to understand a few of the concepts leading to long sales cycles. However, the agencies seemed like a perfect match so we decided to keep them as the core customer group and rotate the product use cases for all our future experiments. In our case rotating customer groups was easier when conducting experiments, simply by replicating the same experiment with different customer segments.

Tweets on twitter @acondurache