Next Steps for Data Visualization Research
Given its youth and interdisciplinary nature, research methods and training in the field of data visualization are still developing. So, we asked ourselves: what steps might help accelerate the development of the field? Based on a group brainstorm and discussion, this article shares some of the proposals we found most promising. We hope that other researchers will join us in ongoing discussion and experiment with new approaches.
Background
Though the practice of data visualization stretches back for centuries, concerted academic research on the topic is relatively young. Two watershed moments include the publication of Bertin’s Semiology of Graphics in 1968 and Tukey’s Exploratory Data Analysis in 1970. These works were followed by early research on interactive statistical graphics (such as the PRIM-9 system) and then, in the 1980s, by graphical perception research.
In terms of the current research community, the IEEE Visualization conference first met in 1990, joined by Information Visualization in 1995 and Visual Analytics Science & Technology (VAST) in 2006. These conferences simultaneously convene each year at the IEEE VIS meeting, with articles published in the journal IEEE Transactions on Visualization and Computer Graphics (TVCG). Visualization research papers also regularly appear at human-computer interaction conferences (such as ACM CHI and UIST), statistics conferences (JSM), and other venues.
Visualization research contributions include new visualization techniques or software systems, the results of controlled human-subjects experiments or qualitative studies of visualization use, and theoretical treatments of visual representations, interaction techniques or design processes. Published work typically draws on research methods from one or more parent disciplines such as computer science, human-computer interaction, statistics, cartography, and perceptual and cognitive psychology.
Adapting the Publication and Review Process
Provide reviewer guidelines. Currently, the primary sources of guidance for authors and reviewers at IEEE VIS are the paper submission guidelines and reviewer ethics guidelines. While both are valuable, more systematic criteria regarding specific contribution types (such as techniques, systems, and controlled experiments) might help enforce rigorous, constructive reviews. Analogous to medical practice, reviewers might benefit from “checklist” aids, organized by contribution type, to help ensure necessary pre-conditions for publication are met. At the same time, it is important to draw attention to known reviewer biases; examples include fixating on shortcomings over positive contributions, over-weighting easily fixable flaws, under-valuing novel but “unpolished” ideas, and letting initial impressions (whether positive or negative) unduly affect the scope and rigor of subsequent review. Annotated examples of both “good” and “bad” reviews could serve as valuable guides, especially for fledgling reviewers in the field.
Craft more targeted review forms. At many conferences, a review consists of a single numerical rating plus textual commentary. However, journal reviews often involve a number of more specific ratings relating to the soundness of methods, quality of writing, and so on. We might reconsider the design of review forms to help ensure comprehensive reviewer attention. Building on the previous proposal, given a contribution type with systematic review guidelines, the review form might include a checkbox a reviewer must click to indicate that they have read and considered those guidelines.
Publish the reviews. Once accepted, only the final revision of a paper and its supplemental material are published. The content of peer reviews and discussion is visible only to the authors and reviewers. This hides from public view valuable material regarding both the contributions and shortcomings of the published work. Publishing the reviews of accepted papers (maintaining reviewer anonymity) would provide context for assessing research contributions, generate example reviews to learn from, and raise the level of reviewer accountability. As a first step, authors might opt-in to published reviews, as now allowed by Nature Communications.
Accompany articles with editorial statements. Short of making all reviews public, published papers might be accompanied by a public summary review. This statement could highlight the research contributions that motivated acceptance, along with identified shortcomings or disagreement among reviewers. Statements could be curated by the primary committee member responsible for a paper (who is already tasked with writing a private summary review) with oversight by the papers chairs.
Require necessary supplemental material. Video figures or interactive demos are commonly included alongside a paper, as static images fail to convey an interactive experience. Benchmark studies require access to the systems and environments tested, and have received increased attention in computer systems research. For controlled experiments, the backing data, stimuli, analysis scripts and specific task instructions are critical. Sometimes these can be reasonably described in the paper text, but often not. To foster replication and more substantive peer-review, we might institute more formal requirements around supplemental material. A healthy and growing trend is to use online repositories to provide “living” supplementary material, which can be shared, copied and extended over time.
Promoting Discussion and Accretion
Public discussion forums. Discussion of research papers actively occurs at conferences, on social media, and within research groups. Much of this discussion is either ephemeral or non-public. The community might benefit from a shared forum for research discussion, safeguarded with a modicum of neutral editorial oversight. To facilitate ongoing discussion, in-person questions after a conference presentation might explicitly transition to the online forum, with a provided URL at which the conversation can continue. A scribe might seed the discussion with a record of what was said at the conference. The end product could be living, annotated proceedings.
Templates to structure critique. Many people are often hesitant to share critical feedback, in part to avoid alienating others. Structured templates may help people formulate comprehensive, constructive critiques. The Stanford d.school advocates an “I Like, I Wish, What If?” format to encourage both positive and corrective comments, as well as forward-thinking extrapolation. Might we use an analogous format to scaffold research critique, which might better engage students in the process?
Response letters to journals. A sometimes overlooked part of the research literature is response letters, which place critique into the research record. For example, Becker et al.’s classic piece on “Dynamic Graphics for Data Analysis” is immediately followed by a commentary from John Tukey. Early issues of the Human-Computer Interaction journal contain a fascinating back and forth among pioneers of the field. Published commentary, vetted by editorial or peer review, could be re-instigated in the field.
Replication and meta-analysis. “Discussion” should also play out across multiple research publications. Insufficient effort is done to replicate and verify the results of prior studies. We could be producing and publishing more of this work. Once topics have received sufficient attention, meta-analyses could help consolidate the field’s understanding.
Research Methods Training
Promote a core curriculum. A number of universities include a research-oriented data visualization class and related classes in HCI, statistics, and experimental design. However, some universities may lack such courses, or students may fail to take them. Developing a core curriculum for data visualization research might help both cases, guiding students and instructors alike. For example, recognizing that empirical methods were critical to multiple areas of computer science, Stanford CS faculty organized a new course on Designing Computer Science Experiments.
Catalog online resources. A core curriculum could be reinforced with a catalog of learning resources, ranging from tutorials and self-guided study to online courses. Useful examples include Jake Wobbrock’s Practical Statistics for HCI and Pierre Dragicevic’s resources for reforming statistical practice.
Provide tutorials. Of course, we should also be providing appropriate methods tutorials in our home organizations and at conferences. Existing tutorials usefully cover tools (e.g., D3) and visualization design topics (e.g., color design), but rarely concern fundamental research methods training. The first step is to recruit high-quality instructors. The second step is to promote the tutorials so that advisors and students are made aware and motivated to attend. Tutorial materials should also be made available online so that those who can’t attend in person can still benefit.
Seek help. Interdisciplinary research requires interdisciplinary expertise. As needed, we should seek out collaborators with complementary skills to help ensure world-class research. Some university departments staff “help desks.” Here at UW, one can get free consulting on topics including statistics, design and data-intensive science. We should take advantage of such resources.
Recreate prior work. One of the most powerful and immediate things a new researcher can do is replicate prior work. Interested in a specific technical area of visualization? Start by re-implementing techniques described in papers. You’ll hone your skills, gain in-depth knowledge into the techniques, and perhaps spark new research ideas. Interested in conducting human-subjects studies? Replicate (and potentially extend) prior published work. Platforms for deploying experiments online make this relatively easy. You’ll quickly learn if the paper you are replicating faithfully communicates the information you need to reproduce the study. You will be forced to work through each detail of both the design and analysis of the experiment. This practice will help prepare you for future independent research. Plus, a replication might lead to unexpected or more nuanced results!
Going Forward
These proposals are just some of the ideas we discussed, and are not intended to be comprehensive. Do you agree or disagree with any of the proposals above? Can you think of other proposals you believe the field should consider? Which steps should we prioritize and implement going forward? And, critically, how might we evaluate their impact?
Please share your comments with us, here or on Twitter (@uwdata). We want to thank brendan o'connor for his contributions via Twitter and the Gordon and Betty Moore Foundation for their support.
This post was collaboratively written by the IDL team.
