Andrew Gelman on the art of asking the right questions
His books include Bayesian Data Analysis (with John Carlin, Hal Stern, David Dunson, Aki Vehtari, and Don Rubin), Teaching Statistics: A Bag of Tricks (with Deb Nolan), Data Analysis Using Regression and Multilevel/Hierarchical Models (with Jennifer Hill), Red State, Blue State, Rich State, Poor State: Why Americans Vote the Way They Do (with David Park, Boris Shor, and Jeronimo Cortina), and A Quantitative Tour of the Social Sciences (co-edited with Jeronimo Cortina).
Next to that professor Gelman writes for his highly influential statistics blog. And it’s an huge honour to interview him for I love Experiments.
Finding out what causes something is pretty much the holy grail of science would you agree? In one of your essays on Causality and Statistical Learning you discuss 3 basic questions in science,
- What happened?
- What might happen if I do X?
- What causes Y?
Why do you think it’s important that people understand the difference between these questions?
They’re just different sorts of questions.
For example, in politics we might want to know how rich people and poor people vote (1), we might want to know what happens if a new candidate enters the presidential race (2), and we might want to understand why white people in the south have such different political attitudes, on average, than whites in the rest of the country (3).
One of our conceptual insights was to realize that this last sort of question — What causes Y? — is ultimately a query about some anomaly in the world, something unexplainable by certain existing theories, as we discuss here: http://www.stat.columbia.edu/~gelman/research/unpublished/reversecausal_13oct05.pdf
Do you like what you have read so far? Get a quarterly update of what I am busy with.
If I look at the current practices in organisations the real questions people are busy with are:
- What do I think just happened?
- What do I think might happen if I push this button?
- What do I think causes our success?
- (Will my colleagues believe me when I tell them my answers?)
More often that not intuition takes the place of data to base their decisions on.
Although I agree on the difference between the inferential questions, I also see the basic need of more data. So if organisations use data gathered from answering just one of the first 3 questions to inform their decisions it would be a big improvement.
What is your take on this?
I think your questions above are fine, and that the next step is to refine them. For example, if you want to ask, “What do I think causes our success?”, you’ll first want to do some descriptive inference: What have been your organization’s recent successes and failures? And then: What aspects of these successes and failures have been surprising? If you planned a certain course of action and it was a success, do you feel that requires an additional explanation? Maybe yes, if you planned other things that did not work. And so on.
The point is that Why questions typically do not get definitive answers; rather, they prod you to look deeply, to understand why you’re asking why in the first place.
Lukas Vermeer (responsible for Experimentation at Booking.com) is promoting the idea that for a “Data Scientist” (I know…) it’s essential to ask the right questions. I completely agree. But that goes for all scientists of course.
A/B testing is a big thing nowadays. But people rarely stop to think what questions they are asking.
“Giving the average response to alternatives, which alternative should I use for the whole group?”, might be a discription of a normal A/B test. “Giving the information I have right now on this individual how big is the chance that she will act on a specific alternative? And how can I learn if that individual wouldn’t be more susceptible to act on another alternative?”, is more a Bayesian type of reasoning applied in machine learning technology, right?
This is a peek into the machine learning application my last company made:
How do we make sure we ask the right questions?
In A/B testing I think it’s important to consider that treatment effects vary by person and situation: what might work for one person in one setting can be ineffective or even counter-productive for someone else in another setting. This should motivate you to fit models of varying treatment effects, what we call interaction models in statistics.
Bayesian reasoning can help here because, once you start looking at interactions, your data will quickly begin to seem sparse. Bayesian inference does partial pooling so that in data-rich settings, your estimates will be strongly data-based, and in data-sparse settings, your estimates will be much closer to your fitted model.
Coming back to asking the right question. I predict that the more organisations refine their questions, they will be more interested in Bayesian Probability. Giving the data you have right now, what can you predict on some individual’s s next move and how sure are you about that prediction? I guess that comes closer to the truth than a frequentist’s approach.
Furthermore as an organisation I would put my money on becoming better at the question:
What might happen if I do X?
Because as an organisation you are about to do something, and you are only interested in how you can affect the outcomes of your action in a for you positive direction.
How do you comment on this more or less coherent piece of proza?
I wouldn’t quite say, “you are only interested in how you can affect the outcomes of your action in a for you positive direction.” You also want to know what might hurt, so you can avoid that course of action. And you also want to know what will have no effect, so you don’t waste time and resources on such things.
Overall I prefer a continuous approach, in which any decision is recognized to have many effects, and you don’t simply try to characterize it as a success or failure.
