Fitting Curves: Only With a Little Help From My Computer
Understanding people’s ability to fit bell curves to data.
If data visualization is to become an all-purpose statistical tool, capable of every phase of analysis from data clean-up, to descriptive statistics, to published results, it’s going to take some work.
Classic statistics is confirmatory in that it requires the analyst to formulate their questions first and then testing them against the data using statistics. Famous statistician John Tukey launched the current era of data visualization when he turned this approach on its head by advocating exploratory analysis: looking at the data first to find patterns to hypotheses. He then argued for switching to confirmatory statistics when testing those hypotheses. His reasoning was that if an analyst creates a hypothesis through one method (viz), and then confirms it with another (confirmatory statistics), we might avoid circular reasoning.
At the heart of traditional confirmatory statistics lies a collection of (mostly) bell-shaped curves. In Tukey’s day, drawing such idealized curves was a slow, point-by-plotted-point ordeal. Today we have tools that can draw normal and other curves as easily and accurately as prior generations could draw straight lines with pencil and ruler. Does this mean bell curves and all the statistical inference they facilitate might now be a tool of data visualization? Might two different visualization approaches, one to explore data and the other to confirm hypotheses, be enough to avoid circular reasoning? In other words, can the data viz actually become the statistical test?
If it were going to be, us humans would need visual intuitions that allowed us to understand with a glance what the curves are meant to represent. In recent work, we conducted an experiment designed to find out if people have those intuitions. As it turns out, it looks like our intuitions might need a little help.
Experiment and Findings
We asked people on an online platform to look at a sample of data drawn from a normal distribution and fit a normal curve onto it; see the figure below. They used a slider to move the middle of the curve (estimating the mean), and used the ends of the slider to adjust its width (estimating the standard deviation). We included four different chart types in this experiment: bar charts, dotplots, box plots, and strip plots.
On average, our 117 respondents did pretty well finding the middle, less well with the right amount of spread. The chart below summarizes this data.
Percent errors were small in placing the centers of the curves, and tightly clustered around 0%. But errors in finding the spread were far more, well, spread out. Even worse, with both the bar and dot histograms, people tended to oversize the curve (errors above 0%), while with the box plot, they tended to make it too narrow (the errors are below 0%). Strip plots yielded estimates with less bias, but no additional accuracy.
Arguably, despite the downward bias in spread errors (which we can correct for), people did best with the box plot (invented by John Tukey himself). This may be the most telling experimental result.
We drew our box plots around 5 points — the median (which for normal data like these is a close approximation of the curve center), then the 10th, 25th, 75th, and 90th percentiles. In other words, these graphics show only precalculated values — a very close approximation to the entirely calculated ideal curves we asked people to fit. Among the four visualizations we tested, they required the least intuition to use.
Implications
If the graphic that leans least heavily upon our intuitions does best, does this suggest our intuitions aren’t up to the task?
We know lots of instances in which our visual intuitions fail us — just ask a data visualization guru about pie charts or look at an Escher print of monks walking an infinitely upward staircase. Perhaps fitting idealized curves is just one of those tasks computers do better?
Of course, this is only one experiment. It might be flawed, or the results might not hold up. And it may still be useful to create a visualization tool that shows users statistical tests in graphic form. If nothing else, it might expose what is otherwise a black box of equations that many people follow like recipes without fully grasping the meaning of the idealized models they are applying to their data. Furthermore, visualization might still make traditional statistics accessible to our intuitions. But if our results prove informative, and if they do hold up over time, it seems like people’s intuitions will benefit from a little help from their computer.
