When speeding up means slowing down
Lessons on running thoughtful growth experiments
by Constance Wellman and Shirley Yang
True or false? To be effective, growth teams have to always be working rapidly: iterating, shipping and moving to the next “win.”
At Dropbox, the Growth Design Team has learned that while this statement is often held as the ultimate doctrine of growth, there are times when slowing down is not only necessary but streamlines efficiency and findings.
1. Spend time getting to know your user
Growth teams tend to start by comparing two versions of an experiment (A/B tests) to determine what experience is most successful. But running a full-length A/B test takes time to design, build, and collect results.
Our Growth team recommends starting with research and following a traditional design process: discover, ideate, refine. This will help you understand your user’s goals, motivations, and needs. While it seems like this process requires more time up front, it will help you be more thoughtful in your approach to experimentation.
How did we do it?
We’re Constance and Shirley, a designer and researcher embedded on Dropbox’s Growth team. Our sub-team’s focus area is the user’s first experience with Dropbox, right when they are deciding which Dropbox plan is the best fit for their needs.
Like most growth teams, we started by rapidly shipping experiments with the goal of building on those that were most successful. Over our first few months on the team, we ran a handful of different experiments, which didn’t result in any conclusive results. We were back at square one.
After talking with our team, we realized we didn’t have any knowledge about our new users and their intentions when they come to Dropbox. We decided to pivot away from running more experiments and instead conduct a series of qualitative interviews. We realized the way we framed our plans didn’t resonate with users. We then used these research findings as a basis for future experimentation.
Before we launched more experiments, we also went back into research to help us determine the best design solution. Our team’s typical process is to run multiple design approaches as an A/B test. Instead, we ran a quick round of usability testing on UserTesting.com to identify which approach resonated most with potential users. We had a decision on design and copy within a week, saving us time that would have been otherwise spent building out an A/B test.
2. Bring in your cross-functional team early
It’s tempting to skip this step because it can be a pain to coordinate everyone’s calendars. Resist the temptation! Bring in cross-functional partners early to call out potential technical or business blockers. This will save your team significant time and money in the long run.
How did we do it?
When we were kicking off our ideation phase, we hosted a brainstorm session. We asked for at least one person from each of the following roles to attend: engineers, product managers, growth managers, product marketers, customer support, design, research, and analytics. We encouraged everyone to contribute their different perspectives, and, most importantly, identify any potential blockers. The solution that came out of that brainstorm was one that would be a positive user experience, not technically difficult to implement, and a large business win.
3. Take time to learn why an experiment was successful
Traditionally, growth teams learn from their losses and iterate to make future experiments better. We believe it’s vital to learn from your successes. If you can understand why an experiment was successful, it will help you create stronger experiences in the future.
How did we do it?
We ran an experiment designed to improve on our goal of helping our users choose the best Dropbox plan for their needs. By all measures, this experiment was a success: it exceeded paid start rates by 3x and had an 88% completion rate. However, when we debriefed internally, we only had hypotheses on why it had performed so well. We sent out a follow-up survey to better understand the root of what made the experiment successful. The survey findings also gave us a qualitative metric to use as a benchmark, so that we can track our continuous improvement on how we guide new users to different plans.
Now that you’ve mastered the thoughtful approach, let’s learn when to be quick!
1. Don’t wait for the perfect solution
While our team was focused on running experiments, our company was also developing a segmentation model that would predict which plans are the best fit for certain users. Rather than waiting to execute once the model was developed, we decided to run a series of experiments aimed at this exact goal. The intention behind this was that any learnings are good learnings. We decided to test a more manual approach, where we asked our users a series of questions about themselves, to inform a plan recommendation. This provided learnings we could act on, while we waited for the model to be developed.
2. Don’t be afraid of running parallel experiments
If you’re working with a large enough traffic size, try to run parallel experiments. Doing so will help you learn at double the speed, and you don’t need to wait until the first is complete before running the second. Just make sure that you understand why one might be more successful than the other, so you can build on it in the future. Our team decided to run parallel experiments to move quickly. We then came up with a cohesive solution that took into account both experiment learnings.
3. Leverage existing components to save time
At Dropbox, our team always tries to design thoughtfully. However, when a team needs to move fast, component libraries can be their biggest asset. Building new components takes additional time and resources. Our team leveraged existing illustrations for our first experiments. This gave us time to evaluate the success of our experiment before investing the time in creating new illustrations.
Now it’s time to experiment!
We’ve found success on our Growth team by knowing when to move quickly and finding those moments to move more thoughtfully. Utilizing both has helped us build more successful experiments. This has resulted in a better, more cohesive experience for our users.
We’d love to hear from you! As you think about making improvements to your process for the new year, what are some ways you’d like to move thoughtfully? When are times that you have intentionally slowed down? What did you learn? Share in the comments below!
Thanks to Andrea Drugay and Brenda Buchanan for the feedback and Justin Tran for the illustration!
Want more from the Dropbox Design team? Follow our publication, Twitter, and Dribbble. Want to make magic together? We’re hiring!
