TL;DR
Data rarely gives you an insight into visceral customer reactions; and hence is a poor tool to figure out where to spend your time and what to prioritise. Developing softer qualitative insight can build intuition, empathy and eventually conviction. Great products are built by teams with a huge conviction for what matters to their customers; and who are prepared to move seemingly unmovable barriers to achieve customer goals
The Experiment Factory
There’s a school of product development — that product management is all about experiment throughput. The thesis is as follows:
Every idea is equal and no-one has any idea what’s going to work; Hence the approach should be to brainstorm lots of ideas and test them all. She that is able to test the most things, will learn the most and their conversion/retention/acquisition rates will improve and they will accrue riches.
I do see this “throwing spaghetti at the wall” approach for product development frequently with startups. It’s never a source of sustained growth. Think about it, if you want to double the rate of growth of a startup in 6 months, then you will need to find experiments the compound value of which is 100% in 6 months. Given most tests will take 2 -4 months to run and get to significance — you’ve got two shots really to make a change that would materially move your conversion rate. The key is less throughput — but figuring out what could move conversion rate by 50%.
The experiment factory approach, doesn’t guide a team towards learning, the problems you need to solve for customers and what really matters. If your learning after a year of tests is: green buttons, sort by price, and pictures matter — I don’t think you’ve learned anything about your product; but have the managed to re-discover established principles of user interface design.
Conviction based product management
Laura is one of the Product Managers at Transferwise responsible for our invite a friend program. We work with quarterly goals and plans at TransferWise and I asked her what was her plan for the following quarter:
She replied, “I want to test everything !”
“Everything ?”
“Yup — subject lines of emails; email content; landing pages; invite a friend rewards… everything.”
“Why ?”
“Because I have no data on what matters.” She responded.
Well that was never going to happen — we chatted about it and she decided to set herself 2 weeks to figure out which ONE thing mattered most to our customers and could potentially have the biggest impact. So she started by talking to customers, active users (and inactive users of the referral program) understanding what frustrated them; pitching them ideas, developing prototypes and watching their reaction
For the quants on the team — this process looked flawed for a number of reasons:
i) Tiny unrepresentative sample size
ii) Laura had many implicit biases which could impair her ability to impartially conduct the research
The small sample size was undoubtedly true, but prior to the talking to customers, Laura reviewed wider data (NPS results; Customer feedback) to understand where to probe. In this case the key she found was helping customers understand how much they saved drives recommendation. She only needed to hear the same thing from a few people in her group on the topic, to know it was relevant for the same reasons to a wider customer group.
The implicit bias question was much more of a real risk with this approach. This means you go into the conversations looking for signals that reinforce what you have already mentally committed to, rather than trying to learn what is the right outcome for your customers. Part of the answer here is to be incredibly self-aware of your biases going into the conversations; the other is to be disciplined in not having a “solution” in mind when interviewing customers.
The interviews helped her in a few ways
i) It narrowed down the “problem she was solving”
ii) It helped understand — why ? — now when she launched a test, she understood the user worry / behaviour she was trying to impact. Tests deepened her knowledge of these domains and the results guided her towards further areas to explore
From Laura and other product managers at TransferWise, I realised Laura’s approach worked for her because of her natural strengths and hustle. The same approach may not work for other product managers. That said every product manager that I’ve ever seen be successful, has always had some way of interacting with customers and figuring out how to use softer quant insights to inform where to focus.
So what’s the role of data ?
Given the above, what’s the role of data ? Well two fold. Data can tell you where to look, where to focus, where to start digging; and secondly data is the only way to validate that the intuition and hypothesis you developed in those customer interviews is accurate. Launching split tests validates the impact of the intuition that was developed in customer interviews and also is invaluable step in the process of hardening that intuition into stronger conviction.
Team conviction
Something quite magical happens — when a team have been working together for a long time on the same customer problem; they develop a shared intuition and a strong conviction on what matters. This is pretty awesome to see.
As the team’s conviction gets stronger, they are able to commit to longer multi-quarter projects — that will significantly move the needle for customers based on the conviction they’ve built over the years.
Some teams even get to the point where they are comfortable releasing features that they cannot easily measure the impact of; as they just KNOW it’s the right thing for customers. There is a huge value in a team getting to this point as the teams are biased to release changes that they can measure. There are some product changes that are very hard to split test the impact of. Some of these types of changes can be hugely valuable — think “magical experience” type stuff — and the only way for a team to commit to these kinds of changes is to have built iron-clad conviction that this is important.
In my experience the changes that a team with strong conviction are able to advocate and execute have the opportunity to materialy move a product and hence a growth rate. To some extent building a strong product organisation is all about building a culture which encourages these kinds of changes to happen.
Note: All pictures from TransferWise summer days. If that looks like fun, we are hiring
