Experimenting towards a goal, the giffgaff way
Experimentation is a journey, not a destination and you might bump into happiness on the road. I decided to share our learnings throughout this journey and how our team has helped in shaping giffgaff’s approach to experimentation.
Be glad you failed and now have time to succeed
One thing that we have learnt in our team is the importance of thinking about experiment failures as learnings, or better yet “time gifts”. The result of not having wasted all that time, building that big thing that wasn’t going to work.
Use small input to prove the likelihood of big outcomes
It is important that the impact of failures is as small as possible, our approach to achieving this has been doing a team exercise to decide which is the smallest input we can use to prove or disprove what we assumed to be the outcome of a bigger solution we want to build.
In practice, it is a pretty simple approach yet effective. Knowing which is the hypothesis we want to test, we have a diverge-converge session where all kinds of ideas are welcome and based on risk vs impact we choose the least risky with the highest value.
Physically, the decision process we use looks like this.
On a practical example, we had a hypothesis which was that if we could build a microservice to recommend alternative phones to members whilst they were browsing, this could help them decide on what phone to buy.
The usual approach would be to simply go on and build the microservice, but by using the above approach we found an easier way to test our hypothesis by simply manually hardcoding an AB test with the same recommendations that the microservice was going to give us.
Whilst, we had a positive response from members, the outcome wasn’t as big as expected which led us to celebrate the gift of time from not having wasted weeks or perhaps months building the microservice.
Align all your experiments to goals
Whilst experimenting is fun and has given us a formula for faster learning, anything can be experimented upon and you might lose your north in the process. This is why it has been particularly important to challenge ourselves to confirm which metric we expect to move if our experiment is successful.
On our board, this looks like this.
We make various hypotheses on how we can move each metric and tie various experiments to each of them. This is how we achieve the alignment of many experiments towards a single goal.
Keep it simple
Many of the experimentation books out there are aligned with each other and provide usually context-less examples of clever experiments, like every Product Manager’s favourite Wizard of Oz when Amazon Echo was tested. Before building the smart speaker we know today, the Echo team simply had a human “wizard” in the next room typing all the responses for a not yet very intelligent speaker to give back to the user.
The reality is that not all experiments need to be super clever or sophisticated and the highest probability is that the leanest ones are going to be simple but can provide the same expected outcome or lead indicators of failure/success.
Release value as soon as possible
When possible design experiments which can be left in place once successful, whilst you build the solution from your hypothesis.
Not all experiments are directly aligned to the same metric
There will be some instances when your experiments are directed at proving or disproving your riskiest assumption and this may not necessarily move the metric tracking your goal but it will derisk your way forward.
One size does not fit all
This is a snapshot of the way we do it at giffgaff, and whilst it may be useful to adopt parts of it, the context will vary from company to company and it’s best to add your twist to it.
Start by starting
The best way to learn is to get on with it, don’t spend too much time trying to design a process. Experiment your way into creating your process and you will see it organically coming alive.
Let us know how you get on as we can always learn and perhaps improve our approach from your experiences. Nice one.
Originally published at https://www.giffgaff.io.
