Dr. Growthlove or How I Learned to Love the Metrics
Sadly, my FinTech company, Vouch, now drinks nectar in the Elysian fields for startups. Vouch issued consumer loans whose terms improved when friends and family “vouched” for you. The obligatory post about our mistakes will be a topic for another time. Right now, I want to cover something we did well: we were good at prioritizing and moving metrics, and I’m going to share the process we used. Before going any farther, I want to acknowledge Rajveer Singh Tut, Aayush Iyer, and Ivy Xu from Vouch’s extraordinary Growth team, without whom this post would have no substance.
Our Growth process worked extraordinarily well for us. I remember a board meeting early on where the board looked at our engagement numbers and said, “You really can’t call yourselves Vouch with a 12% engagement number. That’s a problem.” Using this process we identified 22 experiments to run. 6 of them were successful, moving engagement from 12% to 90% by the next board meeting. Needless to say, the board was impressed, and I’d point to this process for how we did it.
What is Growth?
What the hell a Growth team is anyway? Growth VPs will all answer that differently, but they all agree on 2 things:
- Growth sits at the intersection of Marketing, Data Science, and Technology. Depending on the team, Growth may include Design, Product, or Strategy functions as well.
- Growth is a data- and hypothesis-driven function.
If you are interested in Growth, you can learn all about learn flows, engagement ladders, user accounting, “a-ha” moments, funnel optimization, and such from the masters; Josh Elman, Sean Ellis, and Andrew Chen are the guys to follow.
Metrics-driven management
Vouch’s Growth team was really, really data-driven. The charter was to identify the most important metrics for the business and move the ones that needed help.
In retrospect, I realize that our process was much more about managing through metrics than it was about growth, meaning that you can use this process to manage metrics from anything — marketing metrics, product metrics, financial metrics, etc.
Our process had 7 steps; internal slides describing our process looked like this:
Step 1: Strategy
Starting your planning process with the big picture helps everyone relate their work to the larger purpose. When I think of how we kicked off Growth planning at Vouch, we frequently returned to our vision, which never really changed.
We always wanted to improve financial access. We believed that people build trusted relationships over their lifetime, and that this trust counts when making lending decisions.
Although the vision remained steady, our strategy for achieving this vision changed frequently as our company evolved. Early on, our focus was on proving that vouching was a behavior that prospective borrowers would engage in. Then we needed to prove that that behavior actually enhanced creditworthiness. Then we needed to prove scalability, then unit-economics, etc..
The point here is that you should expect your strategy to change, and with it, your success metrics will change as well.
Step 2: Metrics
Once you understand your strategy, you can define your success metrics. We used Ishikawa diagrams to break high-level metrics into actionable sub-metrics. First start with your strategy’s success metric, for example, unit-profitability:
Then break that into sub-metrics:
And keep going….
The more detail you can go into, the more actionable your brainstorm (next step) will be. After you’ve gone through this exercise, do the research and find out how each of these numbers are performing. It will help step 3 to know if there are benchmarks or interesting trends around specific sub-metrics. You won’t optimize all these metrics right away, but knowing these numbers will help narrow your focus.
Step 3: Brainstorm
I like to bring in the experts and creative geniuses from across the organization for these brainstorms. Although any sub-metric is fair game for discussion, I’ll usually highlight the sub-metrics where the industry benchmarks or internal trends suggest that we have plenty of room to improve. It’s easy to spark a lively discussion by noting, “Our top-of-funnel web conversion rates are 20% below what we know our competitors have. Here are the step-by-step screen shots. What can we improve?”
The output of your brainstorm should be a list of ideas and the sub-metrics you expect each idea to move. Note that a good brainstorm will likely generate ideas that span different functional areas. A goal of improving acceptance rates for Vouch loan offers, for example, could include changes in underwriting criteria, marketing targeting, or product messaging.
Step 4: Prioritize
Once you’ve generated a big list of ideas on how to improve a few key sub-metrics, you’ll need to prioritize. We found an old open source pairwise project and adapted it for our needs. If you aren’t familiar with pairwise evaluations, it’s essentially Hot or Not for ideas. It looked a bit like this:
Every idea that came out of the brainstorm sessions was evaluated on 2 dimensions: ease of implementation and impact on the metric we were interested in. Everyone at Vouch was invited to participate in these pairwise evaluations. Once we had enough data, we could put together a 2 x 2 that identified the low-hanging fruit.
Step 5: Experiment
With a prioritized list of ideas, we would construct hypotheses that could be proven true or false. Team members were free to pick a hypothesis near the top of the list and run an experiment to test it. Our morning stand up ritual involved running through this list and reviewing the status of experiments.
Step 6: Measure
The Growth team was zealous about collecting and using data for experiments. Almost nothing ever ran without some sort of A|B control — even ideas we were sure would be slam dunks. The reasons were:
- We were constantly surprised by the outcomes.
- It was incredibly motivating for a team member to see the concrete, positive results of their work.
I can’t underestimate the importance of good infrastructure for tracking experimental data. We had surprisingly sophisticated systems at Vouch, capable of running concurrent tests and evaluating up-and-downstream impacts through multivariate analysis. We collected enough data to test virtually any hypothesis that anyone could come up with.
Step 7: Implement
Our experiments were often messy affairs. While the Growth charter was to move fast and learn quickly, a successful conclusion for any experiment was to go back and implement the feature or learning in a more sustainable manner.
Wrap-up
If you do have the opportunity to implement this process, I’d love to hear how it goes!
