Experimenting without losing your soul
How I learned to incorporate and embrace experimentation in the design process
Experimentation is a tool in your toolkit
Four years ago I landed on the growth team at Pinterest. The teams I worked on mainly focused on acquiring and onboarding new Pinners and businesses onto the platform. Most of these experiences are extremely optimized for conversion and activation performance and require a tremendous amount of experimentation. I learned quickly how data and experimentation could fit into my design process, but admittedly it was a bit overwhelming at first.
Designers sometimes worry that experimentation means getting trapped in an iterative design spiral that doesn’t allow for new innovative work or that design quality suffers. It doesn’t have to feel this way! Integrating experimentation into the design process can empower you to not only to make really smart design decisions but also a way in which to communicate with cross-functional partners in the right way to gain trust and credibility.
While experimentation on a growth team goes hand-in-hand with design, I believe having an experimentation mindset can be applicable to all different types of product teams, and can be a powerful skill to master as a designer. Here are some things I’ve learned so far and hope they’re helpful to you:
The value of experimentation
When deciding to run an experiment you should be able to complete this statement pretty quick:
Because we launched________ , we learned________.
We can start to tease out how people are behaving and interacting with the product to help us understand the why.
Why is a person taking this action instead of this other action…why was this type of education less effective than the other? For example, we could test two different ways of teaching people how to take an action — one has a pulsar with supporting text and another with just the pulsar directing someone to take the action. If the education paired with text works better, we can infer that people need more written education to understand what we are asking them to do.
Another way experimentation can be valuable to teams is through opportunity sizing — it can provide a quick gut check to see that teams are moving in the right direction or investing in the right idea. For example, a team might be accountable for onboarding people onto Pinterest. The team might want to test specific product education in different parts of a flow to see which is most effective.
Let’s say they test education before and after a person signs up. They could first test this experience on desktop and learn where the education works best. The team can then scale this learning to other platforms like iOS or Android. By running an experiment on one platform first, they know where to invest their efforts instead of testing all three platforms at the same time.
On the flip side to finding successes and opportunities, it’s also crucial to learn from experiments that don’t perform as intended. I’m often humbled by a design not working the way I expect. But these failures can be clarifying, and it’s important not to assume that it’s because it’s a better or worse user experience. Data is reflective of what a person is doing, not what they may or may not understand. More on how to make decisions with data later.
When to experiment
There are typically two ways of thinking about experimentation when designing, and figuring out which route to take can be hard. Here are a couple of questions that might help when determining if experimentation is helpful:
Does the team need to iterate on existing features?
These experiments generally follow the classic A/B experiment structure where the team tests different design variations against the existing control experience. For example, two different button placements on a surface, copy iterations, or two types of education upsells. You can usually decide whether or not something is good enough to ship based on a single experiment.
Does the team want to ship something new?
You might be thinking, “Hey, you can’t always measure the impact of a totally new design” and you are right! There is a big difference between A/B testing experimentation and step-change experimentation. An example of a step-change could be a brand new product surface or feature — like adding a search functionality to the product.
Experimentation can be used for both scenarios, but how you would use the data would be different. When A/B testing, the feature is likely smaller and the team might have tested two approaches, having a clear decision based off of the results. When there are step-changes in the product, experimentation is most valuable to monitor the overall impact that feature has on important company metrics. Product decisions for this type of change likely take into account many factors including how the company wants to move forward directionally, which sometimes can’t be measured or won’t show results in the short-term.
How to use data to make design decisions
Alright, you’ve experimented…now what? I use this quick checklist when determining the path forward.
Did we hit our success metrics? Yes!
Whether you are able to read dashboards yourself or if you have a product analyst supporting your team, it’s crucial to know what success looks like. Establish clear guardrails that the team is or isn’t willing to cross with a project ship. Find what you were originally trying to measure and see if your experiment had an impact. If the experiment has positive results, then it’s likely you should ship it!
Did we hit our success metrics? No… 😩
Bummer, but there are still things to learn here! The data might suggest unintended metric outcomes or show that a desired action wasn’t completed with the tested design. You can use this information to decide whether to shut down the experiment or iterate on your design or content. Oftentimes, the design might need to be changed slightly. Or you might need to approach it from a new angle to find success.
Qualitatively test (when possible) to confirm hypotheses
It’s important to not get caught in a data-only decision-making process, experiment data is one piece of the puzzle. With data, you can infer the way people interact, but it’s important to acknowledge when to use other tools like qualitative research or directional strategy in a ship/no-ship decision. For example, if a team is testing a completely new onboarding flow, qualitative research is helpful to determine if someone actually comprehends the product better with the new design.
Experimentation data is a powerful tool that can help you better understand how people interact and the quantitative impact of the experiences you’re creating. Knowing when and how to experiment will not only help you to create better experiences but will also increase your impact on your team and at your company. Hopefully, by getting familiar with the experimentation process, you’ll find that it’s not actually soul-crushing to a designer at all — it’s empowering!
