Exploring Product Operations: Experimenting Fast and Slow

Intro

There is a common pattern found in product companies called “Running a Feature Factory”. Most of the time engineering teams produce new features without a time to reflect on the progress, while product teams firmly push for new ideas, usually coming from upper management or competition. Though it’s considered an anti-pattern, there is still a chance for feature factory teams success. That chance usually occurs when someone asks the stubborn question, “Why?”

But this is only the beginning. At the core of the “Why?” is the belief that the added value you produce must benefit the end user of your product to be considered valuable. And here, the second question arises, “How do we know that we’re making something valuable?”

It’s always easy at the beginning. If you see certain metrics go up: conversion, clicks, returning customers, revenue, you name it, there is something valuable in what you do. Either it works out and you continue growing the business, or it fails and the game is over. It really works great at the beginning of the journey when you don’t have much to lose. But when you’re already running a healthy business and looking for growth, things change dramatically.

We need a tool to help us out while pursuing the growth of our product. Something easy enough yet powerful, an indicator that says what’s wrong or right.

Experimenting

Experimenting and one its form in particular, called “Randomized controlled trial”. It’s a golden standard when you’re trying to understand what works and what doesn’t, while avoiding the risks of blowing things up quickly. In the IT product world randomized controlled trials mutated to the next best option called A/B tests. We can be totally sure that this feature that we’re working on is the best thing that ever happened to our customers, but when the A/B test shows the outcomes of our experiment and the numbers don’t match, this feature dissolves into oblivion. Right?

Not so often. In the case of very passionate product teams, people usually dive deeper:

• Let’s run the experiment one more time;
• Could the QA team double check the feature;
• Let me go over the whole flow myself, QA team surely missed a flaw;
• Please, show me the numbers once again;
• It simply couldn’t be that way, I feel there is a mistake here.

That’s a natural way of how we become attached to our ideas. This bias is further confirmed by the amount of effort put in all preparations. Of course, there may be a problem with our feature:

• Its “footprint” is too small, our users just missed it;
• Another feature caused a conflict with ours and led to a problem;
• Clients use our feature in a way that was not intended.

Or the experiment itself:

• Adverse effects from the outer environment, like connection issues;
• Wrong target audience;
• Too little of experiment power;
• Problems with A and B buckets split.

Though all the things mentioned above could perfectly fit as an explanation for our failure with this feature, we still don’t know for sure. Or do we?

Ideal Process

In the ideal world, our feature does its job perfectly, and we’re confident in our analytics department that ran the experiment. Each feature is launched into a fraction of our audience to early spot a potential problem, then gradually increased to a 50/50 split and turned on completely if it is proving viability. In a very ideal world, it is turned off in the end, no matter what to be turned on again after some time just to be sure. We collect all of the experiment data, cross-examine it further with activity events, heat maps, session recordings, user feedback and pack it up in a beautiful presentation. Ok, nice, we nailed it. But after investing so much time into this one feature we have about a 10% chance of having a positive impact on our metrics¹.

And here comes the unavoidable prejudice every experimentator has: I need to show progress, and I need it fast. In the scientific world, there is a battle for granted while in the product world, it is a battle for influence, career ladder, and credibility. So, what options are left?I need to power up the experiment machine and throw in features quickly. And here we are where we began: feature factory.

Second to Ideal Process

Though it may seem like an unavoidable dilemma, there is a second to ideal process. This process consists of the following steps:

1. Establishing trust;
2. Working out strong hypotheses;
3. Conducting exploratory tests and/or research.

The foundation is built upon trust. Top management <-> product team <-> analytics team <-> engineering team. Everyone understands that no matter how confident they are, the proposed idea does not guarantee success. Yes, even that one is 100% proven by the relevant data. This seems pretty basic, but is very hard to implement in practice, where corporate politics plays its role. Let’s continue.

Way too often, we come up with weak experiment ideas, for instance: let’s introduce this new user interaction mechanics because we saw it somewhere in the competition. Ross Nazarenko has a wonderful article about this topic. We need to formulate well-thought-out hypotheses that are aligned with our strategic goals and based on the proven facts that we know about our customers. Those hypotheses now need to be compared and planned. The same way we treat our features when we talk about prioritization. Their analyses need to be properly designed by the analytics team. Obviously the number of these hypotheses you can run, say in a quarter, is fairly limited. And yes, there is no guarantee for success.

To build up trust while everyone is busy working on powerful hypotheses we have the last tool in this article: exploratory experiments. Those should be straightforward tests, limited in scope and dedication. Treat them as cheap, easy sensors you spread all over the product to notice userful anomalies or signals. They don’t need to have a strong hypothesis underneath, but be nicely targeted toward a specific problem. You can conduct those experiments really fast and still they are better than running a feature factory. Chances are that a successful exploratory experiment might be later converted to the hypothesis.

Suggested usage of different types of experiments

Final

To sum up, rather than viewing every credible product idea as a Willy Wonka golden ticket, view it as a bus ticket towards learning more about your customers. You shouldn’t (and practically speaking can’t) always use the travel agency on your journey. Sometimes down the road you should opt for cheap and easy rides that boost your traverse experience.

[1] Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, p. 112