Why Big Data Needs Experiments with Dan Simons
Dan Simons is a professor in the Departments of Psychology, Advertising, and Business Administration at the University of Illinois. His research explores the limits of our own minds and the reasons why we often are unaware of our limits.
He teaches classes in the field of visual cognition at the undergraduate and graduate level, and he also teaches introductory psychology.
So to answer your question Dan: What’s PTTRNS about?
I hope Science Rockstars can fulfill a larger role one day in Business Understanding of Science. My cousin Bas Haring (Dutch) also holds the “Richard Dawkins” professorship at Leiden University: professor of Public Understanding of Science. I want to do for business what he is doing for the general public.
In that sense there are a couple of topics important for me. The scientific method and experimentation in general. Common knowledge and theory of mind versus scientific theories of mind and behavior. Biases and ways to improve our decision making. Tricky statistics and of course the ethics of all mentioned before.
In al honesty, what do you think of the quality of experiments that are done by scientists? And how could they, and their business counterparts improve on that?
The category of all scientists is a bit too broad for an evaluation. Even within psychology, there is a lot of variability in the quality of experimentation. For the most part, experiments in psychology are well designed to tease apart conflicting predictions.
That’s particularly true for basic science experiments on cognition (thinking, memory, attention, perception). That said, I and my colleagues have identified some major shortcomings of intervention studies, those designed to test the effectiveness of a treatment group against a control group.
The studies are not necessarily poorly designed, but they tend to neglect controls for some important confounds (see our recent paper in Perspectives on Psychological Science and some of our critiques of video game training interventions).
One major concern about the quality of studies is the use of underpowered designs — not including enough participants to obtain a stable estimate of the size of the measured effect. That’s a shortcoming that psychology as a field is beginning to address.
Such underpowered studies are less likely to replicate and are more likely to produce inflated estimates of the size and importance of the underlying effects. When coupled with some questionable statistical practices that are all too prevalent, there are concerns about the reproducibility of findings.
That’s one reason I and many of my colleagues have been pushing for the publication of direct replications and also promoting the use of pre-registration of designs (just like clinicaltrials.gov does). There appears to be a groundswell of support for these sorts of changes to the publishing culture, and that makes me more confident about the future of the scientific literature going forward.
These are science-wide issues, but psychology has been taking the lead in implementing changes. I think the same issues apply in business-related experiments, many of which are conducted by psychologists in business schools. I don’t think the issues they face are any different, although they are often more intervention focused given the applied questions involved.
Do you like what you have read so far? Get a quarterly update of what I am busy with.
In your opinion, is there a way to cultivate a culture of experimentation? And if so, how would we go about doing that?
For me, the main way to inspire a culture of experimentation is to identify alternative explanations. If people think through the alternatives, they tend to see the need to distinguish among them. The danger is that it’s all too easy to lock onto our preferred explanation and miss the obvious alternatives.
Scientists are trained to think about and test for alternative explanations, but they too can miss the obvious ones. It’s not an easy problem to solve, but once you start thinking in terms of alternatives, experiments are the natural way to address them.
Running experiments is hard as it is. But what do you think of using difficult to understand algorithms to cluster relevant data and making computers tell us what patterns that data, aka the Big Data promise, reveals?
Big Data can address some problems, but not all. It allows you to uncover relationships you might otherwise miss, and it might suggest new strategies or further experiments. But, there’s a danger too — it’s easy to find patterns in noise, and if you base a strategy on those relationships, your strategy might be based on the noise.
We need experiments to determine whether those associations are causal.
What are your personal role models on experimentation, perception and public understanding of science? (you can’t answer Chabris on any) How can we learn from them?
My personal role model for scientific thinking is Ulric Neisser. He inspired my “gorilla” study and his approach to thinking about science and cognition influence how I view the world. I wrote about him here (he recently passed away — a huge loss for our field).
Someone showed me a commercial the other day that mentioned attention bias and then told the public that they should pay more attention to motorcycles… Which I think was a quite a painful example of how science gets misunderstood and misused. And this is even benign, imagine when people want to misuse it.
I’ve actually written about the “watch for motorcycles” campaigns. Although it might help to alert people to motorcycles, people are great pattern detectors. Unless they encounter motorcycles regularly when they drive, the signs are unlikely to help much because we learn to look for those things that are typical.
In The Invisible Gorilla we wrote about some studies showing that bicyclists and pedestrians are hit more often in places where they are less common. If you see pedestrians all the time, you look for them. When they’re rare, we don’t.
Do you have a vision on how scientists can influence understanding of their research?
Yes. They can write about it themselves. Surprisingly few popular books about psychology are actually written by the researchers themselves. Most are written by journalists. Although many do a good job, others are unable to evaluate the research well enough to separate the wheat from the chaff.
That’s hard even for trained researchers, but at least they but they are trained to situate findings into the larger literature and to evaluate the quality of the research methods and statistics. If more researchers wrote for the public, they would better convey an accurate impression of what the science shows.
That said, many researchers are not skilled at writing for a general audience. We need to train students and faculty to do that. I and some of my colleagues teach graduate courses on speaking and writing for general audiences for exactly that reason.
Finishing up. Not all readers might know you are the guy behind the famous Invisible Gorilla (attention bias) study. What would you say is now the most worrisome invisible gorilla in business? And how to tackle it.
I’m not sure that there’s one obvious gorilla lurking in business — business isn’t a monolith, and there are invisible gorillas at every level from management to sales to accounting. To me, the bigger issue is not that we miss things. That’s true for all of us, all of the time.
What concerns me more, and what I think has a bigger impact, is our mistaken belief that we will notice anything important. That failure of intuition can affect business decisions across the spectrum.
Do you like what you have read? Get a quarterly update of what I am busy with.
Originally published at www.sciencerockstars.com on February 6, 2014.
