Introducing product experimentation — “it would be nice to test that…”
Introducing experimentation to a product which has previously done none, is exciting, but also challenging. On a smaller or newer site (both of which were true here), the ability to adapt quickly is key. That very same fact, can make for braver and interesting experiments, as there are less established norms and loyalties.
I won’t retrace the goal of product experimentation, many smarter people than me have covered this and many great articles exist about why product teams should use experimentation (see ‘useful reading’ section). It can be hard to move (virtually) without seeing how many experiments the tech giants run.
What I hope is that most product managers want to be able to experiment in some capacity, where creative hypotheses are validated scientifically with a/b (or other format) tests, and where ideas are tested without committing to expensive development blindly.
The learning curve of trying to build an experimentation velocity is sharp, and these ‘teething experiences’ have provided some good learnings along the way, 4 of which I’ve delved into below.
(1) A good experiment involves a lot of people
Our product is firmly in the publishing world, and by its nature, content is king. It’s been really important for us to make sure the editorial, marketing and commercial teams are involved to ensure the quality of the content is maintained.
There are many permutations where a marketing campaign can scupper an experiment, content changes have to be restricted, or proper copywriting for an effective experiment is required. Early hopes of just getting into the interface, making a few cheeky changes and reporting my winning findings were dashed pretty quickly.
Learnings:
- We’ve found getting everyone excited and involved in the value of experimentation has really helped us embed it into our process. Providing proof for why we’d like certain changes gives everyone confidence in the process. Crucially, it means people are willing to dedicate time to help us, which is no insignificant ask. We’re in a place now where people are constantly requesting new experiments, which is a nice culture to have.
(2) Your OKRs (or other WHY based format) should remain the reason you’re selecting experiments
In the Product world, when working with the wider business, conversations about deliverables rather than outcomes are inevitable. Scattered feature ideas and multiple requests are par for the course.
Framing these ideas around your Key Results first, I.e. do they relate to them at all, then using experiments as a way to prove or disprove the idea’s hypothesis, gives clear communication lines to stakeholders about why you’ve prioritised certain items when it comes to development. It can also help move the conversation away from feature delivery dates, and to progress against key results.
Learnings:
- When discussing any candidates for product development which will move our OKRs, we include discovery and post launch activities as a rule. These are now heavily book ended by experiments we plan to run.
- We have made a commitment for one key result workstream to always experiment first. Using selected ideas which we think will have the biggest impact on our conversion rates. This means we can get ideas in front of users sooner, as we don’t let perfection be the enemy of the good. If it works, we can productionise it.
(3) Waiting for statistical significance can be a fruitless endeavour
Possibly a little more controversial and for some sites this might not be a problem, but we’ve run many experiments where we would be waiting an eternity to hit a calculation for statistical significance. Not all features, pages or sites are blessed with high footfall. It also can be a hindrance to smaller products who, by their very nature, need to adapt quicker as they try to find that market fit more rapidly.
Learnings:
- We take a pragmatic approach. Our regular reviews mean we can check in on active experiments, if we see trends are holding strong (fig 1), or similarly, nothing of note is materialising, we’re not afraid to make a call.
- For these cases, we have to be careful not to put too much weighting on findings which haven’t reached significance. We’ve found iterating on these themes we believe to be successful have helped us gain more confidence as we test it from slightly different angles.
(4) Consistent process — I can almost hear the sighs! — really helps
Setup is tough, prep is lengthy, and reporting and decision making is a team job. It’s also worth noting that a good process can take time to become embedded. If you don’t have a robust pipeline of experiments (and a good reason to do them — re: linking to OKRs) you can easily (and we did) end up with weeks or months between running your experiments. Or worse, rushing out ideas without proper hypotheses which link to your goals.
Learnings:
- Having a regular meeting with the experiment experts in our company has really helped us break the back of some tough problems, plan and help us analyse results correctly.
- Documenting is valuable, but keep it readable and visual. We found a distinct lack of investment (emotional, that is) if you have a spreadsheet with 20 columns full of numbers. I still have that doc, but we also have a deck with a one-pager for each experiment that we have all approved. This is very visual, explains what we’re doing, why, and with big traffic light themed results. Be sure to add fireworks for the greens!
So there you have it, a few learnings from a smaller site who needs to adapt quickly, trying to embed experimentation into everything we do. Persevering with the setup and process demands has allowed us to be in a place where we have confidence that the development time we spend is on improvements that are going to have a meaningful impact. Saying “it would be nice to test that”, is no longer a throwaway phrase, it’s our first port of call.
I hope you find this useful, and if one of those points resonates with you, then that’ll do.
If you’re interested in seeing the product i’m referring to you can check it out here: https://www.thetimes.co.uk/money-mentor
Useful Reading
https://medium.com/@jseiden/to-write-better-okrs-use-outcomes-e82be6e7b460
https://www.mindtheproduct.com/how-do-you-implement-effective-product-experimentation/
https://www.fastcompany.com/3063846/why-these-tech-companies-keep-running-thousands-of-failed
https://www.amazon.co.uk/Experiment-Driven-Product-Development-Data-Informed-Approach/dp/1484255275
