Experimentation Over Experience in Product Culture
By Stefan Bala, Project Manager, Cognizant Softvision
The importance of experimentation and tips for success
A couple of years ago, I had the opportunity to venture into the realm of Product Management. Our team’s directive was to improve the user experience on existing features of a product and add new ones born through client interviews. Having a delivery background, I had some knowledge on Product lifecycle, but I worried I didn’t have enough experience as a Product professional.
With every release I was afraid to embrace failures and didn’t take advantage of the opportunities to invest in learning. Being a “numbers person,” I was so focused on outputs and forgot all about the outcomes. Leadership was supportive in allowing me to experiment or take small risks on ideas I had that could enhance the user experience. But when I went back to the whiteboard I got stuck on where to start and how to experiment.
At this point in my career, I believed that domain seniority and experience were the most valuable assets. I felt pressure since I wasn’t as experienced as I thought I needed to be. With no degrees or certifications in Product, it was difficult to integrate and understand the product vision and strategy. Still, I forged ahead in an effort toward professional growth.
Luckily, I was surrounded by people who worked the lattice left and right and they knew from the beginning that they were not looking for a product manager with past experience. Their point of view was that experience rarely matters, considering the speed at which the technology and market evolves. Instead they were looking for someone who was highly adaptable, obsessed with information sponging, and who could come up with creative solutions to problems independently.
As Fareed Mosavat, former Director of Product at Slack wrote, “Good experiments advance product strategy. Bad experiments only advance metrics.” I soon realized that great products are born from experimentation. The following outlines a successful example of an experimentation process we followed on our team.
Step 1: Define your North Star
Time is of the essence, so our team recognized the need to create a blueprint for product experimentation in order to maximize value and efficiency. The first step was to define our North Star, which defines your business goal for the sake of experimentation.
We brainstormed in multiple workshops on what the metric would be for our product. There were numerous questions and premises we tried to answer in order to understand what “good” looked like for the client’s business. What business goals did we want to achieve? What was the outcome we expected?
Once we reached a consensus on the North Star metric we knew what we needed to focus on. Since a considerable amount of data is produced with each experiment, we need to know exactly where to look so that we could learn from it.
Step 2: Tailor an experimentation flow
Establishing an official process for experimentation was the second step, and it helped us a great deal in achieving an experiment flow that we could visualize and continuously improve. Inspiration came from Agile design thinking and lean methodologies — why couldn’t we have a ceremony tailored for our purpose? Visualizing the experiments was borrowed from one of Kanban principles: make your work visible to yourself and others, while the experiment flow was a loop of testing and adapting fast, celebrating failures and improving the business hypothesis.
The secret here is to make this a repeatable process while keeping stakeholders in the loop on how we are pivoting, persevering or even killing business ideas. A significant advantage to this approach is demonstrated through the removal of organizational silos, since “to drive a culture of experimentation, collaboration is critical.”
Step 3: Get the story from the data
Gathering and monitoring data was part of the action plan created for what should be done with our results. This third step helped to assess if the result was positive and if it could be polished for scaling the experiment. And in the case of a negative result, we would ask ourselves why it failed. Where did we go wrong? Should we go back and validate them once again?
Either way, we needed to implement the results in our test hypothesis or create a new one in case the data exposed a different “story.” This is what builds a culture of experimentation.
7 Tips for Experimentation
There are tons of experimentation strategies out there, you just need to find one suited for your product’s vision and be able to scale it up. In my opinion, experimentation is the opposite of following a framework, so instead of outlining steps for building a resilient culture of experimentation, I’ve compiled some helpful tips:
- Learn to embrace failures and celebrate them, as there might be 10 failed experiments for every single one that succeeds
- Surround yourself with people hungry for data and the latest trends on the market. Offer them a great opportunity to prove themselves and earn a seat at your team’s table
- Cultivate curiosity and creativity so people can think independently and come up with ingenious solutions to problems
- Be a dreamer, go stargazing, find your North Star metric and share it across the company, aligning everyone on the importance of using it to build assumptions and hypotheses
- Get different perspectives by encouraging experimentation and foster a culture of continuous innovation
- Build an environment of trust where anyone can express their ideas without fear, and reward scrappiness even though it may lead to a failed experiment
- Know when to stop experimenting. Limit the number of ongoing experiments, follow your instincts and be frugal as there are times when you need to put customers’ value above your experimentation desires.
Lastly, always remember– try low risk changes through trial and error, fail but fail fast and leverage all those results in your learning process.
Sources:
