Visualizing Variable Veracity: Visualization Papers At CHI 2019
Everybody seems to enjoy Twain’s popularization (or invention?) of the epigram “there are three kinds of lies: lies, damned lies, and statistics,” but I was always more of a fan of his “get your facts first, and then you can distort ’em as much as you please.” The latter phrase seems more relevant to visualization, where designers have so much control over what data to collect and how to show it, for better or worse. We’re just extremely good at using data and charts to fool other people (and ourselves). This year at CHI I saw quite a few papers pushing back against this impulse.
ACM CHI (Computer Human Interaction) is a yearly conference that showcases a wide variety of work from all areas where humans and computers must coexist, from robotics to games, sociology to design. This year it was held in (suspiciously) sunny Glasgow from May 4th-9th. Visualization is just one relatively minor part of the CHI community, but there’s been enough visualization papers the past few years to fill up my schedule.
We’ve written about a few of the papers that were at CHI already for this blog. Evan Peck, Sofia Ayuso, and Omar El-Etr wrote about their (best paper award winning!) work “Data is Personal: Attitudes and Perceptions of Data Visualization in Rural Pennsylvania.” And I took a crack at summarizing my paper “Ethical Implications of Visualization Research.” So I won’t be covering those any more than we already have. And of course I have neither the space nor the patience to talk about all of the visualization papers that were presented at CHI. But I’d like to talk about a few papers that help us to be more reliable, robust, and overall trustworthy with data.
Increasing the Transparency of Research Papers with Explorable Multiverse Analyses
Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, Fanny Chevalier. Paper / Website
Way back when the words “replication crisis” were just beginning to cross into the mainstream, statistician Andrew Gelman wrote about the problem of the “garden of forking paths.” Even before we get to the variability in the data itself, researchers have so many potential decisions that they can make when they perform their analyses that we have to treat any particular experiment as just one particular point sample of a whole garden of potential experiments (with perhaps radically different outcomes) that could have happened.
Despite this variability, and the high degrees of freedom present in research, the way we report research is through static, limited, and non-interactive papers. This work, which received a CHI Best Paper award, seeks to address the mismatch between academic papers (which are only one peek at one set of choices) and this wider space of potential analyses.
They solve this through the use of interactive papers (here’s an example). These have the form of a standard research paper, but with hyperlinks that allow you to cycle between different analysis choices. The figures and text of the paper alter in response to these choices. The discussion and conclusion, however, do not, encouraging authors to write “defensive” discussion sections that can “survive” across many possible worlds of analysis.
I think this work is interesting both because it offers much needed support in my desire to skip the replication crisis in visualization, but also because it shows that the way that we communicate research is in desperate need of an overhaul. There’s a mismatch between the way that we do science, and the way that we communicate science.
A Lie Reveals the Truth: Quasimodes for Task-Aligned Data Presentation
Jacob Ritchie, Daniel Wigdor, Fanny Chevalier. Paper / Website
Visualization lacks a central dogma, but one of the things that people have generally agreed on is that you should probably start your y-axis from 0. If you don’t then you’re creating a misleading graph. Yet, as Vox pointed out, there are many cases where starting from a value other than 0 is not just fine, but arguably a better and more communicative chart than the version that “tells the truth.” Both types of charts seem to have the capacity to mislead. For every “If Bush Tax Cuts Expire” there might be an equal and opposite “The Only Climate Change Chart You’ll Ever Need.” How, then, to navigate this apparent conflict in guidance?
This work uses the GUI concept of the quasimode to allow viewers to navigate between different types of charts, alternating between a focus and context (or distorted and undistorted) presentation. You have to explicitly invoke the other view (say, by holding down the mouse pointer). Once you stop, the view animates back to its original setting. I like this idea much better than just creating a dogmatic guideline about whether or not to always exclude or include the zero: it gives viewers context, but not at the expense of hiding important changes.
How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation
Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q. Vera Liao, Casey Dugan, Thomas Erickson. Paper
Decision-Making Under Uncertainty in Research Synthesis: Designing for the Garden of Forking Paths
Alex Kale, Matthew Kay, Jessica Hullman. Paper / Website
One of the things that I’ve tried to push back against in the past few years is this idea that data just sort of spontaneously appear in the world, that datasets are the result of some sort of naturally occurring process that collects objective truth about the world, and the job of analytics is just to harvest or extract those reservoirs of truth. In fact, data are collected, structured, and organized by people, who have a great deal of control over what the final product looks like. It’s why adages like “data is the new oil” have never sat well with me: it’s a resource, sure, but it’s not like oil: it’s more like a novel (an authored thing) or a car (a designed thing). The main similarity to oil seems to be that people are paying a lot for it, and that if you mishandle it then you can devastate entire coastal regions.
Both of these papers report on just how much decision-making and control the people in charge of creating our data have. In the first paper Muller et al. looked at data scientists and data workers at IBM, and in the second Kale et al. looked at scientists performing literature review and research synthesis. In both cases they found that it’s not just mining that results in data, but strategizing and deciding and even creation. Michael Muller put a particularly dramatic spin on a quote from one of his participants in his talk: “I am the ground truth!”
A key difference in these papers is in what they inspire me to do. Muller et al. makes me wish we spent more time on the human-centered parts of data science: I know quite a bit about what kind of code data scientists write, but not very much about how they think, and how their decision-making is instantiated in their final products. For Kale et al., there’s the explicit acknowledgment that each one of these decisions introduces doubt, uncertainty, and risk in my final analyses. How can I be sure that we did the right thing, and that we made the right choices? What are ways of surfacing these doubts and risks to the people making decisions with these data, not just the architects of a particular dataset?
VizNet: Towards A Large-Scale Visualization Learning and Benchmarking Repository
Kevin Hu, Snehalkumar “Neil” S. Gaikwad, Madelon Hulsebos, Michiel A. Bakker, Emanuel Zgraggen, César Hidalgo, Tim Kraska, Guoliang Li, Arvind Satyanarayan, Çağatay Demiralp. Paper / Website
This last paper is less directly connected with risk and validity, but it threw me for a loop all the same. VizNet is a massive dataset of both visualizations and their backing data scraped from many public visualization galleries (including plotly and the dear departed Many Eyes). It turns out that if you have a data source this big you can do all kinds of machine learning magic to do type inference, visualization recommendation, or just plain counting.
It’s this counting that most interests me. In their dataset (which, granted, is biased towards visualizations meant for public consumption), the average dataset backing a visualization had just 17 rows and 3 columns. We talk a lot about big data, but when people are making visualizations to convey understanding to other people, the data we see is very often quite small. Most of the work that goes into contracting, aggregating, and structuring that data is invisible in the final product. Instead of a visualization being a complete picture of what’s going on, maybe it’s more like an iceberg: the visible part is just a small fraction of what’s happening (and the real danger is underneath the surface).
Overall Impressions
There was a lot of strong visualization work at CHI this year; I only touched on a handful of papers, and in a limited way. But it was heartening to me to see people think very deeply about how human beings use data and visualization to structure the world around them, including the places where things can go awry. I still see lots of opportunities for visualization work on provenance, trust, reliability, and persuasion: all the messy places where researchers will have to go out and talk to actual people about the actual things they want to do with data, and where there are real stakes for getting things wrong.
