Conflicts in Lean Experiment Design

Every day when I arrive to work, my team and I ostensibly have a single goal: to prove (or disprove) a hypothesis that we’ve all framed together in the past.

Once we prove it (or definitively fail to), we’ll use that learning to form a new hypothesis on a different (although usually related) topic, and move on to trying to prove or disprove that one. In this way, we gain knowledge of a problem and experience of how to validate that knowledge.

If it sounds like we’re scientists, it’s because we’ve borrowed some words and ideas from scientists. But make no mistake, we’re not the same — and that can make acting like one very interesting.

So…what do you do?

My team and I are software developers. Making software isn’t science any more than two other jobs I’ve done: constructing a home or cooking a meal. While carpenters owe a lot to physicists and cooks owe a lot to chemists, one isn’t the other — the former applies, the latter discovers and describes.

The priorities of each party day-to-day are also pretty different, even though they work with similar materials, and attempt to answer related questions. When presented with something that’s applicable but not understandable, a carpenter will be perfectly happy, while a physicist would find that same thing frustrating, or at least challenging. On the other hand, a physicist would be satisfied with a finding that was repeatable and understandable even if it didn’t imply any applicable processes for the carpenter to make use of.

It’s worth mentioning that, however we might try and style ourselves, my co-workers and I are much closer to the carpenter than the physicist.

So…why do you pretend to be scientists?

In the interest of simple answers first, complete answers next: “Scientific Management” is the name of a business book from 1911, and it’s still cited today in business books like “The Lean Startup”. So, the simple answer is, business owners see legitimacy in science, and an enterprise thereby gains legitimacy with them when it appears scientific. Business owners also pay our salaries. Ergo, in a simple sense, we pretend to be scientists because it earns us credibility with the people who pay our salaries!

To be more complete about that answer, we have to look at why business people admire scientists. One tempting possibility, because it would wrap everything up quite neatly, is this: what if it’s true and business can be meaningfully viewed as a science?

Is there really much of a difference?

As it’s not the purpose of the article I don’t want to take too long to do it, but I want to emphatically state the following in the clearest terms I can: business is not science. Although the disciplines sometimes benefit each other, it comes down to a difference in the definition of success for a practitioner of each.

In science, you’re only as good as your ability to publish your results and get other people to repeat them without your help. The results themselves can be extremely underwhelming — “dropped things fall” isn’t exactly insight, but it is the result of a very important science experiment. If the result reached can be explained to someone and then they can reach that result again, that’s a success.

In business, success is making more money than you spent. It doesn’t matter if you can explain how you did it, or whether you can show someone else how to repeat the process. If you spent $1,000, made $1mil in the end, and didn’t get caught breaking any laws, you’re an excellent businessperson, mostly regardless of how you did that.

One practitioner in this case sets as a target what should happen (profit should happen), and the other sets as a target how it should happen (measurably and repeatably). Hypotheses are guesses at what should happen, but scientists are just as happy with a solidly disproved hypothesis as they are with an apparently proven one.

What kinds of pitfalls result from these differences?

In my career, I’ve seen friction occur on teams anytime that these differing perspectives on the purpose of experimentation aren’t sufficiently recognized or respected.

Pitfall 1: Any Color, As Long As It’s Black

Henry Ford famously said that the customer could have any color of Model T he wanted, as long as that color was black. And, if you develop software using Agile methodologies, chances are the business has come to you at least once with a similar statement: “You can run any kind of experiments you want, as long as the product I’ve already specified is built within the timeframe and budget I’ve already specified”.

This is frustrating to engineers, especially when the businessperson regards it as some kind of largesse that they’re allowing their “teams” to “work the way they want to”, when really they’re requiring two jobs’ worth of work from each person for one job’s worth of pay: Waterfall developer and Agile developer, in one.

However, instead of succumbing to the frustration and assuming the business has evil intent, consider that the person asking you for the unreasonable may not be aware it’s unreasonable.

After all, from the business perspective, money in = product out, not learning out. It’s on you to make sure that you’ve set expectations for outputs ahead of doing any work. That helps avoid this particular pitfall. You may end up needing to educate your collaborators a little bit about what a software engineer is actually for. The bad news for them is that you’re not the code monkey they wanted. The good news is, employed correctly, you can help them craft a better plan than they have today, not merely implement their current plan!

Pitfall 2: Believing in the Old West Main Street

When Western films and TV shows were very popular fare, it was important in the movie industry to be able to believably put characters in an “Old West frontier town” on the cheap.

They considered building realistic replicas of such a town, but this was found not to be scalable. It was too expensive to build one for each movie, and too difficult to suspend viewers’ disbelief if every movie took place in a small set of similar-looking towns (or, God forbid, the same one every time).

Solution: just build the fronts of the buildings, and paint them. Don’t build walls, don’t put anything inside, and don’t film the buildings from angles that show these deficiencies. Only actually construct the buildings that the plot requires characters will eventually enter; even then, maybe construct them on a sound stage instead of outside in the desert.

This worked great at the time, allowing the set builders to scale their work and keep costs down. It just meant that the filmmakers needed to work within an added constraint: namely, don’t shoot a building that isn’t real from an angle that makes that obvious.

When my team and I build software, we’re doing something similar to the Western-set designers: we focus our attention to detail on the parts we think the user will actually notice or that we think , and we build the bare minimum in other areas. It’s very comfortable and fancy to do this when we can apply official-sounding terms like “Last Responsible Moment” and “Minimum Viable Product” to these practices.

But the issue with buzzwords like this is that “viable” is an adjective that’s very dependent on its object — viable for what? Is the product minimally viable to test a given hypothesis, or is it minimally viable for making gobs of money in a consumer marketplace?

Either can be fine; however, if there is miscommunication as to which has been built, you’re gonna have a bad time. If the people responsible for prioritizing work on the product come to believe something is “done” because it worked during an experiment, that can lead to unfortunate awakenings as to how much “more done” it’ll need to be to work in what is admittedly a pretty different situation (e.g. production). When this happens, I like to say that the director started believing in the set designers’ lies — the Old West main street.

Pitfall 3: You Think You Want That, But You Don’t

My third pitfall title comes from Blizzard Entertainment’s famous gaffe: when asked about standing up servers for an old product that users felt nostalgic for, a senior exec went on record with “You think you want that, but you don’t.”

Consumers in general are experts on… basically nothing. They can’t be relied on to do simple things like read, even. However, the entire system is predicated on their being experts in at least one thing: what they want and how to spend their money to get it. Telling someone they don’t know what they want (and you do) simply doesn’t work. You have to show them what you think they want, and then let them tell you whether you’re right.

This comes into play at times when experiments are designed in such a way that disproving the hypothesis is more desirable to decision makers than proving it, or vice versa. Ideally, an experiment will be designed in such a way that if you prove your hypothesis, you learned something that will drive you forward; if you disprove it, you’ll learn something else that will also drive you forward, just in a different direction.

Do not allow experiments to be framed in such a way that they can ‘succeed’ or ‘fail’, because any conclusive result is a scientific success! Design experiments in such a way that any given outcome teaches you something that you couldn’t verify before, and that is in some way actionable.

When feedback isn’t in contradiction or opposition to an assumption of value that the business has made, stakeholders will feel less threatened by that feedback and more willing to give it its due. You won’t have to watch a senior member of your leadership tell a crowd what they want like he’s some kind of God.

Thanks for Reading

So that’s my case: business people want software engineers to behave like scientists for good reasons, viz. exploring demand for a product that doesn’t exist is something like exploring a physical phenomenon for which the explanation doesn’t exist.

The disciplines of business and science do have some differences, though, so it’s important to know your audience even while you’re behaving as a scientist, to avoid conflicts around those differences. Primarily, I recommend you involve the business in experiment planning, do not treat hypotheses as goals to achieve, and don’t allow features of prototypes to be equated with features of products.

Good luck and good hunting, investigators!