Maximizing the ROI of Incentives with Uri
Uri Gneezy is the Epstein/Atkinson Endowed Chair in Behavioral Economics and Professor of Economics & Strategy at the University of California, San Diego’s Rady School of Management.
Forbes (well actually Dan) has named Uri one of world’s powerful new economists in the world.
Gneezy, who frequently contributes to the Freakonomics website, is known for designing simple, clever experiments to demonstrate behavioral phenomena that open up new research directions in behavioral economics. Examples include his work on when and how incentives work, deception, gender differences in competitiveness, and behavioral pricing.
Gneezy and coauthor John A. List have published a must read book on the hidden motives and undiscovered economics of everyday life, titled “The Why Axis.”
In 2014, Gneezy cofounded Gneezy Consulting, a business consultation company that specializes in behavioral economics.
Let’s talk incentives. What incentives do you want to test (or did you already test) to see how organisations should be influenced to start using experiments to inform/steer policy and strategy?
I am interested in maximizing the ROI on incentives. Take the auto industry, which spends tens of billions of dollars a year on incentives. How much of it is “wasted” — i.e., is targeted to people who would have purchased the car even without the incentives? My educated guess is the vast majority of it. And even the incentives that actually promote the desired behavior could often be improved. My goal is to try and understand how to increase the ROI on this investment.
Why are you so sure experiments are such a good thing?
You can’t get the causality right without running experiments. Very often we approach a problem with a given intuition, only to discover that we were wrong, and the incentives actually work differently. People are motivated by incentives, but we often need to run the experiments in order to understand how.
When wouldn’t you use an experiment? (as an organisation)
Think about the media disaster of facebook experiments and others — a company should use common sense to see when experiments might annoy its customers, and avoid such tests. Before running any experiment with a company, I ask myself — what if the media will learn about it, is this going to creat a problem? If the answer is yes, I avoid running it.
Okay, good to know your fundamental ideas on experiments. Now let’s dive a bit deeper.
There is a lot of talk in our discipline on the importance of education in experimental design and statistics. Word on the street is people seem to have difficulties grasping the concept and understanding how to proper execute experiments. Could you comment on that?
If you build a bridge, you hire an engineer who knows the math and physics needed. When people run experiments, they often think that it’s trivial, and they just repeat simple mistakes that experienced people know how to avoid.
For example, people make mistakes in randomizing the participants between treatments or setting the right controls. Running a bad experiment is worse than not running an experiment at all, because it might bias your perception of, for example, what motivates your customers.
Do you like what you have read so far? Get a quarterly update of what I am busy with.
For your academic work you also cooperate a lot with companies. Which companies do you think are leaders in the field of experimentation? Why so?
Many companies learned how to run proper experiments. For example, in my book coauthored with John List (The Why Axis) we give the example of Netflix that lost about ten billion dollars of its value because it introduced a change without testing how it’s customers will react to this change. They have learned their lesson, and are now testing almost every change they are making, including very small ones. The question is why did they have to pay billions of dollars to learn this lesson.
Replication. Yes :) That became quite a thing this year in academia. What can we learn from that as organisations?
Replications are more of a problem with laboratory experiments where issues like power (number of observations) and external validity are important. This is not the case in well-designed experiments in companies in which the question is highly relevant to the operation of the company (problem with external validity) and in which the number of observations is very high.
Do you like what you have read so far? Get a quarterly update of what I am busy with.
Another very naive question. When I worked at Booking.com we ran a lot of experiments and I loved that! But you might have already understood that. We literally ran 50.000 experiments (ballpark figure of course) within the persuasion principle Scarcity in the last years. And at least 50.000 experiments on Social Proof. Different content variations of the principle. But we didn’t do much analysis and learning within the principle.
Running 50,000 experiments is a bad idea for exactly the reason you mentioned. Experiments should be grounded in behavioral hypotheses in order not to shoot in the dark. It is also often the case that such experiments are not the best fit for the question you are trying to answer, and you can’t invest enough time in analyzing and understanding what the data actually show. It is much better to run few well-designed experiments and understand their implications fully.
Cialdini did some follow up studies for example within social proof to understand better which herd we were more likely to follow (people like yourself and so on). Somehow we didn’t do much of this at Booking.com.
Naive intro ended. Now the question comes. Is it possible to have learned from social proof experiment #1, to improve #2 and learn more about the details of social proof to develop further with experiment #3. For example you would learn from #45 how to address in-group out-group stuff and so on? To continuously learn from the experiments, advancing theories by adding more detail.
Why I ask, there is a reason, is that if so, this to me sounds like the competitive advantage you want as a organisation. Okay, you have your own context, and so you are testing your own hypotheses. But the more you can develop existing theories on consumer behaviour, the better your position to stay ahead of your competition.
Do you agree? Am I way too naive?
It is true that this is the right way to proceed. Start with a well-motivated hypothesis, test it and then analyze the results and think about its implications. Often this process will suggest a way to modify your approach, which you should then formulate into a new experiment to test.
For example, in my work with Edmunds.com we have tried to find incentives that will motivate customers to click on ads from dealers. We found that giving incentives in the form of gas cards worked better than cash. We then tried to “price” this difference, that is, to estimate how much money is equivalent to gas card.
We found that $200 in gas cards was more motivating than a $500 discount. Understanding why took some more experiments. Only after we were confident that we understand why, and what is the optimal level, we actually implemented the new incentive.
What is your dream for experimentation’s role in society?
I’d like companies to rely less on tradition and intuition, and more on data. While companies are open to learning from big data, they are often reluctant to use a more targeted way that can expose the causality, and not just correlations.
