2.4 Running Successful Growth Experiments
By utilizing experimentation to help us decide if changes to product, marketing, and operations are effective, we avoid having to rely solely on our intuition. Without experimentation, we would implement changes and hope that we’re right slightly more than half of the time. Rather than engineering growth, we would be relying on an art form, which would be dominated by a select few that had outstanding intuition such as Steve Jobs (or those who claim to have this level of intuition). Growth engineering would be inaccessible to the majority of us.
Experiments make growth accessible to nearly everyone because they follow systematic ways of testing hypotheses to reach specific outcomes and are not unlike experiments in the physical and social sciences. Admittedly, growth experiments are usually not as rigorous as in academia, but the fundamentals are still the same. Anyone that learns the basic experimentation methodology can lead successful growth development at their company. Of course, you will likely be more effective with greater experience and practice, but it’s important to learn strong fundamentals from the beginning.
In this section, we will review the key components of successful experimentation. The first step is understanding what are you trying to achieve with your experiment. Are you trying to improve how long users spend on your site or how quickly the can get their tasks done? Then we will consider how to measure whether or not the changes that you implement have successfully accomplished that objective, or not. Then we will discuss ways to create a sounds hypothesis about ways to reach your objective. Finally, we’ll review the nuts and bolts steps in running good experiments as well as gathering and analyzing results.
2.4.1 Define Your Objective
The first key to effective experimentation is defining your objective. Why are you trying a particular change or solution? Why are you changing the wording on the registration page or removing steps from the checkout process? Is it to improve the conversion rate of those tasks or to weed out low-value customers? It is important to document your objective, so you can hold the results accountable to that criteria. Perhaps you did increase the conversion rate with your product changes, but you let in a bunch of customers on which you are losing money. Was your experiment a success? It depends on your objective.
2.4.2 Focus on One Variable at a Time
Successful experimentation almost always relies on changing one little thing (your variable) to see how it affects a given outcome. The main reason for this is that it’s very hard to reliably change and measure many parameters at once and determine which ones were responsible for the outcome. In other words, it’s very difficult to establish causality when changing many variables at once. Indeed, that is what makes social sciences such as economics incredibly difficult subjects. In the real world, many factors are changing and when social scientists want to understand how a policy affects certain outcomes such as wealth, it’s very tricky to isolate the effects of the policy from other changes that are occurring at the same time.
Growth engineers also have to be mindful of this fact and should seek to change one variable at any given time. For example, you might be interested in increasing the conversion rate of people coming to your website and signing up for your service. Rather than making a bunch of changes to the messaging on your site and the registration process, you should change just one little thing. For example, you might change just the color of the “Register” button and test the results. Then you might change the positioning of the “Register” button. Later you might try labeling the button “Sign up” rather than “Register.”
Likewise, if you want to experiment with ways to improve customer acquisition through marketing you don’t want to change the messaging of your email campaign, while you change the ad copy on your online campaign, and start a radio campaign. Instead, you want to focus on testing just one little thing. Perhaps you think that the messaging in your emails can be improved, so you should just change the messaging and measure how that change affects your acquisition.
This approach also extends to operations. Perhaps you think that by improving the customer service experience, you might be able to improve the retention rate of customers. Rather than switching your customer service platform, retraining your support staff and hiring new personnel, you should focus on one little thing — as little as you can imagine. For example, you might start by testing only versions of automated email response that are sent to customers that submit help tickets.
Once you isolate the variable that you want to change, you can hypothesize how to do so effectively. That will serve as the basis for your experimentation.
2.4.3 Forming a Hypothesis
When I was an economics student, my professors practically etched into my brain “form a hypothesis backed by a model before you start experimenting.” It’s tempting to jump right into experimentation, but I have seen enough people get disoriented because they lacked a hypothesis to guide them to know that you should give some thought into how you’re likely to go from A to B.
Imagine that you are at one end of the city, and at the other end is a big pot of money. You know generally the direction that you want to go, so you start walking because you can’t wait to get your hands on that loot. You take a right turn, and you see that you’re heading in generally the right direction. The road bends a little, so you take a left turn and you come to an impass. You take a few more turns hoping to keep heading in the right direction, but soon you are hopelessly lost. It probably would have been a good idea to write down some general directions and landmarks before you started walking.
Growth science can be a lot like that. Some experiments will clearly take you in a positive direction. Other experiments will clearly take you further away from where you want to go. However, many experiments will be ambiguous, and, worse, they might contradict other experiments that you ran in the past. It’s very easy to get confused if you don’t at least have some general directions written down in the form of a hypothesis. For example, your hypothesis might be: “We can increase the conversion rate of new registrations by reducing the number of steps that users must take in the registration process.”
As my professors said, “form a hypothesis backed by a model.” The second part is critical. Take a look at the model that you’ve developed and identify the parts that are most likely to drive retention, engagement, and acquisition. For each input in your model, list out the changes that are likely going to drive more customers, get them to buy or use your product more, and stick with your product. Remember that you’ll want to brainstorm experiments to your product, marketing, and operations. Your list does not have to exhaustive at this point but should lay out some obvious things to try.
2.4.4 Choose Your Protocol
The second part of a successful experiment is choosing the right protocol to test your hypothesis. Given what objective you want to measure and what changes you are testing, how can you best implement your experiment? As we’ll cover later, the tools that you might have available for changes to a product might be vastly different than the tools that you have available to test changes to operations. Not only do you have to know how you’ll implement a given change, but you’ll also need a game plan for measuring outcomes. Your protocols might be as varied as an A/B test for two variations of your product design to simple statistical comparison of a change to your customer service operations.
2.4.5 Define Success Criteria
Even with your objective in place, you want to be very precise about your success criteria to stay disciplined and ensure that you are only keeping those changes that are actually worthwhile. For example, let’s say that your objective is to decrease the number of customers that abandon your product after reaching out to your customer support staff. Your hypothesis is that by increasing the number of support staff you will be able to provide better support and will be able to increase customer retention. Perhaps you run an experiment where some of your customers are assigned to more support staff and you find that the additional support staff did indeed increase retention of those customers in the treatment group by one percent.
Should you implement this change? Let’s say that a one percent increase in customer retention translates to an increase monthly revenue of roughly $20,000, but you would need to hire ten additional support personnel that would cost you an additional $40,000 per month. Clearly you should not be hiring additional customer support staff based on the results of the experiment and the associated costs. Your success criteria should include that the change not only increases the retention rate but that it is also economically viable. Without a good success criteria, you might find yourself adopting changes that are detrimental to your business.
2.4.6 Conduct Your Experiment Methodically
Running a credible experiment is not trivial, but it’s important in ensuring that your data will be accurate and robust enough for you to make sound conclusions. The first thing that you have to worry about is making sure that you are recording all the data that you need. This might including things like the number of times that users come to your registration page, the number of times that they get through the entire task, how much you spend on a particular campaign, or how long it takes for your customer service staff to respond to help tickets. What data you record depends greatly on what variables you want to isolate and what outcomes you want to track.
The next thing that you want to focus isolating the variable or property that you want to test. We already covered this point previously as you are creating a strategy, but what often happens is that team inadvertently change other factors beyond what they want to test. For example, if your objective is to increase the fraction of people that make it to your registration page and your hypothesis is that by changing the button color on the “Register” button from gray to bright green will help more people notice it and click it, it’s critical that you only change that one thing. However, what can happen is that your designer changes the color of other links or buttons to make sure that they don’t clash with the color of the button that you are testing. In such a case, you are actually testing multiple changes at once, and it will be hard to say if the outcomes were due to changing the “Register” button or the other buttons.
Finally, if you have to run your experiment multiple times to get robust data, you must make sure to keep the implementation the same each time. For example, you want to try displaying larger product images on your online shopping app in an attempt to drive purchases. You do not want to try the change once a product page with something boring like screws and then with something exciting like electronics. You should try to repeat the conditions of the experiment as precisely as possible to get the most accurate data.
2.4.7 Collect Your Data
Presently, there are many tools that do the data collection for you, but there are still far more types of experiments than there are corresponding testing platforms. This is particularly true for experiments involving non-digital marketing and operations such as training and customer support. In those cases, when you cannot rely on a pre-packaged testing tool, you need to make sure that you gather both the right data and enough data to perform a successful analysis. You could fill a library with books written on the subject of effective data collection such as Data-Driven Marketing by Mark Jeffery , and we’ll cover some of the basic metrics to track later in this book. The starting point is really just thinking carefully about what is the outcome that you want to test and what metrics best capture the outcome. It is also important to consider what are all the factors that might affect the outcome and measure those as well, so you can determine if changes in the outcome were driven by your experiment or those other potential factors.
Beyond recording the right data, you also need to capture enough data to make reasonably accurate conclusions. There are many factors that will determine how much data you have to collect, but the most important are the size of the effect and the level of statistical significance (certainty) that you’re willing to tolerate will drive how much data to collect. This is a fairly technical topic and is outside the scope of this book, but the key point is that if the effect is small, you’ll need more data to figure out whether the change that you are testing really did have an effect or if it’s impossible to tell.
2.4.8 Analyze Your Data and Form a Conclusion
There is a whole academic field tied to hypothesis testing and determining causal relationships called econometrics, and to really do causal analysis well requires very specialized knowledge. However, this should not deter you the slightest bit since we do not have to be totally scientific in our approach. We only have to be reasonably sure of our conclusions, which is still superior to just guesswork and intuition. Of course, the more valid your experimentation and analysis, the more efficiently you’ll be able to engineer growth, which can be a great competitive advantage.
Later in the book, we’ll go over various forms of analysis that you can perform ranging from simple correlation graphing to sophisticated techniques such as randomized control trials. What form of analysis you’ll want to chose depends primarily on how certain you need to be of your conclusion. Perhaps as a startup, you just have to be more certain than guessing since you can implement changes rather cheaply. However, if you are a product owner at a large corporation and the change in question requires hiring a large team and costs millions of dollars, you’ll probably need to be certain that your conclusion is right. Time to hire an economist or a highly-trained data scientist.
This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018. Be sure to check back on Monday to learn about iterating on past experiments. New sections of Growthzilla are published every weekday.
