The kill experiment

This article is part of a series about experiments. While the experimental approach focuses on whether something is a good idea by running experiments, the kill experiment focuses on one thing only:

Will users buy it?

The kill experiment is normally part of pretotyping, lean startup, and design thinking, but it’s more focused, as “Will users buy it?” is the core. This is extremely important when working with innovation because here your primary job is to figure out if it will be profitable (more economical terms could be EBIT, EBITDA, ROI or similar). To do this, we need to know if we can sell the idea to customers.

Principles of the kill experiment

To know if customers will buy it (use it), a kill experiment must follow certain principles to work:

• Users have to believe the product/feature exists (users cannot know they are part of an experiment).
• Users have to commit to buying the product/feature (e.g., by giving their email or clicking a payment method).

Why so? The origins of these principles are found in some valuable lessons I learned when studying psychology:

1. There is often no connection between people’s behavior and what they say. So you cannot ask people as you’ll get unreliable answers.
2. When running psychological experiments, participants cannot know what is being measured, as it will change their behavior.

Note: If you run the kill experiment for a feature or change to an existing product, the central question becomes: “will they use it? / will it improve?”

Kill experiment for an innovation: Will they buy it?

A very important time to run kill experiments is when developing an entirely new product — a new innovation. Here you want to know if users will buy the innovation.

When I worked as a Senior Product Manager for Innovations in Citrix, we made a building block exercise, where we came up with a great idea: With the current products we could build, an “Uber for IT support” — drawing on Uber’s business model for our innovation. We wanted to create a platform where users could get IT support instantly, and IT supporters could find users. Together we would create the world’s biggest help desk:

We also wanted an Uber for IT support: we called it “Instant Relief” (working name. Yes it is cheesy)

There are several aspects of this idea worth running kill experiments on, but there are three in particular:

• Would people in need of IT support buy it from our site?
• Do people have an urgent need for IT support and will they use it?
• Would IT supporters use it?

As we already had access to a large group of IT support companies, we initially ran 25 interviews with both companies and individuals. This gave us some confidence that the big question was: Would users would buy it? So to test this, we made the following plan:

Front page of simple website. The button simply starts a chat session.

Let’s build a simple website with minimal functionality. All we need is a website with a chat functionality. If users come to this site, we then pull them over to our existing IT support system manually. It will be an ugly and chunky user experience, but it will work.

Pretend theater: It seems like a huge number of profiles are available, but in reality, only two persons pick up calls for all profiles.

The idea was to create the impression that a big platform of IT supporters already existed by showing a large number of IT support profiles. In reality, we just wanted to hire 1–2 IT supporters that would then pick up all calls.

Besides learning if users would buy it, we were also interested in the best possible version of the idea. For example: Should it be a fixed price like Uber or a variable price like Airbnb? But that was not the only thing. We needed to test each step of the innovation to know if users would buy each of the steps:

A large kill experiment for an entire innovation. The end goal: Will users click a payment method. If we cannot get here, we should not build it. But we would also like to learn if any other steps in the process make the idea fail.

Initially, the main question was if users would start a session and pay (i.e. would they buy it?) But as we didn’t have any paying system, how did we receive payment? Well, we didn’t. If people clicked a credit card symbol, we simply said: “You are in luck. The first session is free.”

Getting users in

How do you get users in to find out if they would use the product? Well, here we would use traditional marketing, as this would probably be the way we’d get users if we did decide to build the product. We could, for example, run a target Adwords campaign in a selected US state.

How did it go?

Well, working for Fortune 500 companies has its disadvantages. One of them is that I cannot tell you. But let’s just say that we were able to make a decision based on the kill experiment.

When to run the kill experiment?

Initially when you have an idea, most likely you conduct some desktop research. This is easy because the cost is very low — but it’s also extremely unreliable.

If you work in IT, I recommend that you run a kill experiment in the earliest possible stage.

Run kill experiments as fast as possible, so you do not get attached to an idea.

User interviews are a good way to gain an understanding of how an idea is perceived, but as a kill experiment, they are quite unreliable. To my understanding, surveys are completely useless as a kill experiment.

If you would like to know more about when to run the kill experiment, read the order of experiments.

The Experimental Approach consists of seven interrelated articles. Click here for an overview of the series.