Validating Experiments
So you’re using Upperstory and you’ve created at least one Experiment (If not read this to get started) you’re probably wonder what to do next. Well the first step, is to run an Experiment. Go out and gather the evidence required and if you’re part of a team get them involved, Upperstory was built to be collaborative. If it’s just you, “No worries, mate!” simply get cracking and start logging your learnings.
Right, so you now have run your Experiment and got some interesting insights, take some time to review it. Does it match what you expected in your “Expected Outcome”? Was the result better, worse? How does that then support your hypothesis?
When it comes to Experiments, there is no right or wrong, the concern is not with was the ‘Task Completed’ but rather does the Evidence support your hypothesis or not.
See this is a fundamental difference between Project Management and Experiment Management, on the outside you could be confused and start pulling similarities. But in Project Management the task is either done or not and in an Experiment it’s either true of false. So you’re being Evidence Driven, using experiments to drive the agility of your decision making, it’s that True or False that will ultimately guide the decision about what to focus on, what to build and ultimately what tasks you need get “Done”. Crucially however, unlike “Done” and “Not Done” if an experiment is Valid or Invalid neither one is better, the point it to discover if you’re on the right track. And sometimes the most value is in knowing sooner rather than later that you’re not.
Valid
This is the green light, your evidence supports your hypothesis. At this point ask your self:
- How reliable was my experiment? If you’re unsure or want further clarification run the Experiment again, try change something small in the test and see the effect. Tip: You can Clone Experiment which makes the really easy.
- What’s the next Experiment you need to run?
Invalid
If an Experiment is Invalid this should be a warning sign. It tells you that your hypothesis was not correct. Now this may be for a few reasons, maybe the test was wrong, perhaps the wrong customer, maybe your questions weren’t right. There could be many reasons but what is for sure is you don’t want to move ahead just yet.
What you should consider doing now, is either putting the brakes on completely and look for a completely new opportunity. Or create a modified experiment to rule out and potential weakness in the Experiment. But then again perhaps with a slight pivot you may discover a customer you never expected or a way to convert you users better you hadn’t considered yet. This is your time to look up and see what opportunities could exist.
