A visualization research story

VIS 2020 was my 10th IEEE VIS conference! It’s also the last year that the three tracks of InfoVis (visualizing abstract data), VAST (visual analytics), and SciVis (scientific visualization) will have separate program committees and sessions. When I saw Jason Dykes’ “love letter to InfoVis” I decided to reflect myself on what it’s been like to be part of this exciting community.

This post is a bit different from others I’ve written: it assumes a little knowledge of the visualization community. It’s a recounting of what inspired me at various VISWeeks, and how I perceived my own role in the community. These memories are a partial view representing what came to mind when I sat down to write this, leaving out lots of other great inspiration I’ve gotten and friends I’ve made.

Beyond feeling called by Jason’s invitation for more love letters to the InfoVis community, I have another agenda in writing this: to reflect a bit on how I see the value of open criticism of research, a sometimes controversial topic with which I have some experience.

2010: In the beginning, there was Salt Lake City

My first time attending IEEE VIS was in 2010 when it was held in Salt Lake City. Hadley Wickham et al. won Best Paper for their work on line-ups, which attempted to translate statistical hypothesis testing into a graphical inference task. I was deeply inspired. Jeff Heer and Edward Segel published their paper coining the term Narrative Visualization. I recall thinking, wow, one can publish papers like this here? I had a paper was for the Telling Stories with Data workshop led by Karrie Karhalios, Matt McKeon, and Joan DiMicco that year, which ended with an interesting discussion between Jock Mackinlay and Barbara Tverksy about the difference between being immersed in exploratory data analysis and the process of recounting what happened. I left with the feeling I had lots to do, and that this a group of people I wanted to be among. During a break in the workshop I talked to Nick Diakopoulos, and proposed to him the idea of a paper on what seemed obvious to me —that visualizations designed to convey narratives illustrated rhetorical choices intended to lead to certain interpretations. We wrote it over the following year and submitted for VIS 2011. It’s now my most cited paper (somewhat ironically given that I thought I was just stating the obvious!)

2011: It was the best VIS, it was the worst VIS

VISWeek 2011, in Providence Rhode Island, was my second VIS. To my surprise, both papers I had submitted were accepted: the Visualization Rhetoric paper with Nick and a paper called Benefitting InfoVis with Visual Difficulties, with my advisor Eytan Adar and Priti Shah. Priti had introduced me to various results in educational and cognitive psych related to how sometimes making materials harder to perceive or interact with could be useful for learning, leading e.g., to better comprehension or recall. The paper we wrote was mostly a literature review but with an argument that was meant to be provocative, criticizing the assumption that visualizations should be minimalistic in design and support performance efficiency, and pointing to various ways that less “efficient” representations and interactions can stimulate active cognitive processing. We cited a fair amount of evidence from educational and cognitive psychology to back it up. However, the overall claim about Visual Difficulties was a bit of a misnomer, as the evidence we cited was often not referring to visual disfluency. A few things we cited didn’t replicate (though not until a few years later). Perhaps because it was ambitious and provocative, it won an Honorable Mention.

I actually didn’t know about the award until the first day of the conference, when I checked the program. It was exciting, but at the time I also don’t even know that I understood the significance of paper awards. I was mostly concerned with doing a good job on my two talks which were nearly back to back in the same session. Both talks went well, and I was shocked at how enthusiastic and encouraging the reception was, even if I got a little pushback on the Difficulties argument in the QA. In my eyes, I was just some random Ph.D. student, without connections to any well-known visualization lineage. I felt incredibly grateful to feel accepted and like my ideas were even appreciated.

There was one weird moment, though, at the banquet, when Nick introduced me to Ben Shneiderman. Ben was characteristically enthusiastic and encouraging, congratulating me on the talks. But I couldn’t help but notice a look of horror coming over the face of someone else at the table as we were talking. After we talked for a few minutes, the horrified person interrupted to get Ben’s attention, gesturing to something he needed to show him on his laptop. He led him away from the table. It was an unsettling moment, because I sensed it was about me. But I had no idea who the person was or how it could be so urgent.

As I was boarding the plane headed back home, I saw an email from a friend, with a link in it, saying something like, you might want to read this. It was a very critical article about my paper, posted on Stephen Few’s blog, going through the argument point by point to dismantle it, and ultimately concluding it was nonsense. I remember thinking, this is the worst thing that could possibly happen to me. I would rather lose a friendship or be dumped by someone I really cared about than have someone attack my work. It felt a bit like the world was coming to take away the one thing I cared about.

In the weeks that followed, a few people in the visualization community wrote blog posts defending the paper, leading to a broader discussion in comments. I was again moved, this time by the fact that strangers were sticking up for my work, but hated being the topic of public discussion. My mentors pointed out that it’s unlikely that someone would write a scathing critique if what I said hadn’t seemed important. So I tried to shrug it off, and even appreciate that I had written a notorious paper. It was certainly not the first time I’d felt like a black sheep; my life had in many ways been a series of experiments of different ways of going against the grain. So I figured I’d try to embrace it.

2012–2015: Moving on

My interests branched out in my third year as a Ph.D. student. I put aside the idea of finding ways for graphs to make people think harder, and started working on uncertainty visualization and automated visualization for news. VIS 2012 was in Seattle, and coincided with an internship I was doing at Microsoft Resarch in Redmond with Steven Drucker, Nathalie Henry, Bongshin Lee and Danyel Fisher. I enjoyed my time at MSR, and my visits to UWashington and Tableau Software to give talks so much that I decided I somehow wanted to start my visualization career in Seattle (which I later ended up doing!) I did the doctoral colloquium and talked about what I was then calling visualization bootstrapping, now hypothetical outcome plots, showing draws from non-parametric bootstrapping across time and space, for complex visualizations like communities in node-link diagrams. I met more people, like Pierre Dragicevic, Yvonne Jansen, and others from Inria, and was inspired by Jo Wood’s work on sketchy rendering, Nadia Boukhelifa’s work on sketchniess as a visual encoding for uncertainty, and Heike Hofmann’s use of line-ups for testing the power of competing designs.

VIS 2013 was in Atlanta. That year I was presenting a paper from my work while interning at MSR on the role of sequence in interpretation of narrative visualization. Needing to present a paper in the main track made me realize that I was less over my 2011 criticism than I thought I was. I wanted to do research and see what others were doing, but I had no desire to get up and share anything of my own. I remember telling Robert Kosara at the conference how I was dreading presenting. He was encouraging, assuring me that the community accepted my work, after all they were publishing it. I very much appreciated these words, since somewhere in my mind I still saw myself as a pariah. I recall getting the talk going fine, and Fanny Chevalier presenting her work on measurement analogies (concrete scales) in the same session with me, which I found inspiring.

In late 2013, I finished my Ph.D., I did a one-year postdoc at Berkeley with Maneesh Agrawala, which was tough to fit in but extremely valuable intellectually. I went on the job market about a week after driving to Berkeley from Michigan, but tried to minimize the time I spent on it (Note to Ph.D. students: don’t do what I did!) I skipped VIS 2014 in Paris, which I recall some people later saying was the best VIS ever, because I didn’t feel like I should take a week off when I was doing such a short postdoc. I recall reading Michael Correll’s work on alternatives to error bars, and Kindlmann and Scheidegger’s paper on algebraic vis, and wishing I’d been there to see the talks and hear the buzz about these.

I did some interviews, got some offers, and took a faculty job at UWashington, where there was a thriving HCI community and arguably the best visualization research community in the U.S., with the proximity of Tableau and Microsoft Research. I joined the Interactive Data Lab, a fantastic community unto itself. In my first year or two as faculty I worked more on projects that were less vis-centric, like generating measurement analogies and dabbling in NLP. I experimented with seeing myself as someone for whom visualization was part of, but less central to my work.

VIS 2015 was in Chicago. I vaguely remember it; meeting Kristi Potter, Steve Franconeri and his students. Maybe I was still getting used to being faculty, or because I was imagining that visualization wasn’t as central to my work anymore, but this VIS doesn’t stand out as particularly memorable to me. I do recall starting to think that visualization should be much broader than just mapping data to visual features. It should be about any means of adding context or helping people reason about data.

2015: More public criticism

Also in 2015, I published a paper on hypothetical outcome plots in PLOS One with Eytan Adar and Paul Resnick. This paper looked at how well people could judge effect size from animated draws compared to static visualizations like error bars and violin plots. I’d struck out trying to publish on the topic at vis in earlier forms, probably because the earlier experiments with complex visualizations weren’t ready, but also had experienced what seemed like knee jerk reactions based on reviewers’ prior conceptions (How can animation be helpful?! Look at Barbara Tversky’s meta-analysis on animation!!) However, visualization’s intersection with statistics had always been the most exciting area to me, and so I kept working on the topic. The paper was met with another public critique from Few. I was surprised (maybe because there hadn’t been any premonitory looks of contempt this time). I remember thinking, Seriously, again?

But this second critique bothered me much less. This time it was an argument against the idea of animated uncertainty visualization and the questions asked of participants, suggesting that people wouldn’t use visualizations to judge effect size information like probability of superiority, since that could be answered by querying the data. My collaborators and I found both of these points puzzling: Few seemed to be arguing against using visualization itself, and against the idea that people sometimes do inference when looking at graphs. (Ironically the same question, about effect size, has been used in two papers that have since then won me awards). This time it also seemed somewhat predictable: here I was again suggesting that the most minimal, static chart design might not always be the best answer, and here I was again getting backlash. This helped put the first critique in perspective. I wrote a public response to this one to make my reaction known.

While I recall feeling much less personally bothered by this critique, I do recall a somewhat depressing realization that no matter how careful I was, it wouldn’t necessarily ward off people taking shots at my work. This realization coincided with a general malaise settling in as I realized that faculty life, at least as a female working on technical topics in a large interdisciplinary department, was also full of chances to be misunderstood or underestimated. I had a paper on automated generation of measurement analogies that I liked get rejected repeatedly around this time, for some reasons I didn’t agree with. I concluded that the only option was to believe in my own work and suppress the drive for external recognition. I recall thinking, this is rock bottom, but it’s a good place to be.

Around the same time, I thought about writing a response about my first paper too, renouncing some aspects of it, reinforcing the parts I thought were valuable. I’d had this thought periodically since 2012, when Andrew Gelman asked me if I wanted to debate Stephen Few on his blog (Andrew had cited me in a paper back then, and Few had spoken up to inform him it was nonsense). I felt like maybe it was finally time to do something about it, since I no longer felt very attached to the work. But mentors discouraged me, and I decided not to bother.

Also around this time, Few criticized a few others’ papers, including another junior female researcher’s work for the second time. The visualization steering committee ultimately responded in comments on his blog, suggesting it was inappropriate. I remember feeling very uncomfortable during all of this. For one, I was still undecided if I thought the critiques were fair or not. I hated the fact that the person writing them seemed to have an agenda unrelated to the papers he was talking about, perhaps connected to the understanding of visualization off of which he made his income as an author and consultant, and that he was a bit of a loose cannon who seemed willing to say anything to get his point across. But I also felt he had a right to say what he wanted, and that he seemed to genuinely care about visualization. And, he sometimes made some good points. Studies I cited in the Difficulties paper didn’t all replicate, for example, and there were a few holes in some claims, which he pointed out in detail. It was hard at times to separate my emotional response to what felt like cheap shots, but I felt the tension in my beliefs.

However, the bigger reason for me feeling uncomfortable during the months surrounding these events was the more obvious one — having people in your field publicly discuss you, is pretty awful, and perhaps even worse the second time around. I remember hating the optics of a situation in which multiple senior male members of the community were having a conversation about several junior women who at the time had much more to lose. I guess I could’ve stepped in myself and commented on the blog, but the online discussion felt at that point a bit like something that had spiralled out of control. I was in my first year as faculty, I had plenty of other things to think about. Writing a public response and moving on seemed sufficient.

2016–2018: My Golden VIS Years

Between 2016 and 2018, I really enjoyed VIS, and continued to be surprised at how the appreciation people showed for my work. VIS 2016 in Baltimore was the first VIS where I remember feeling very busy due to people asking me to speak at various events. I didn’t even have a paper in the main conference, but I must have given three or four talks. I met and had an inspiring chat with Chris Johnson about uncertainty metaphors. I felt lucky to be part of the UW IDL, an amazing set of colleagues, who made the conference more fun. I recall my excitement at Matt Kay’s talk on reanalyzing correlation judgment results for evidence on Weber’s law, in which he reminded everyone of how to do good statistics, as well as my excitement that he was interested in being part of the visualization community. In the same year, at a Daghstuhl seminar I met or got to know better more highly creative people in the community: Fanny Chevalier, Jason Dykes, Tim Dwyer, Moritz Stefaner, Jack van Wijk.

In 2017, despite giving birth to my daughter about three weeks before the VIS deadline I worked on papers anyway, and had three accepted. Zening Qu’s and my work on multiple view consistency won an Honorable Mention. I was also excited about work Yea Seul Kim and I were doing on eliciting people’s priors and predictions via interactive visualization, to see if it could help them grasp concepts like sampling distributions or remember information better. My mom came to Phoenix with me. Mostly I remember it being much better than CHI 2017 or EuroVis 2017, both of which I’d also gone to that year, but when my daughter was only a few months old. On both those trips, I had barely slept at all due to what could only have been post-partum anxiety, maybe an hour per night. So VIS 2017 with a seven month old felt like an accomplishment in itself (Note to new parents: Just skip conferences for the first year. It’s not worth the stress!)

In 2018, in Berlin, I had a few papers, and brought my daughter and my husband. Enrico, Danielle, Robert and I had lunch and decided to start Multiple Views. Draco won best paper, inspiring everyone with the idea that visualization knowledge could be codified and examined more rigorously. I got asked to be on the organizing committee, met with prospective students, gave talks, and hurried from meeting to meeting. I remember recording a “VIS recap” Data Stories episode there with Enrico and Robert, something we did for a few years in a row. Rushing to get to the room from a practice talk for a paper on evaluating uncertainty visualizations I presented that year, I literally ran into a glass wall. Luckily no one seemed to see.

2019–2020: A growing lab, diverging interests

VIS 2019 in Vancouver was as busy as 2016 through 2018. I showed up a few hours too late for Andrew Gelman’s keynote at VDS, and was surprised I didn’t hear more buzz about it, beyond that he had used no visual aids. I recall having dinner with the recently formed MU Collective led by Matt Kay and I, and being excited for Yea Seul Kim, who was getting ready to go on the job market. I liked the paper I presented, Why Authors Don’t Visualize Uncertainty, and a later section in it where I’d started to reason more formally about what happens when people judge “signal” in charts, but I had felt while writing it that it was a bit of a swan song. For awhile I’d been finding my inspiration in economic and decision theory and mathematical psychology, which seemed distant from vis and HCI in values. I sensed my interests were diverging from visualization, or were at least becoming more concentrated around issues of statistical communication and decision-making under uncertainty, which only sometimes intersected with visualization.

Somehow, despite being virtual, VIS 2020 has brought up lots of feelings! Getting positive reception on my keynote at VDS proposing theories of inference in visualization research gave me encouragement that maybe I can bring some of what I love in other fields to vis. Again I was blown away by the sense of community and enthusiasm, and enjoyed watching my students, like Alex Kale, start to become important voices in the community themselves. Jason Dykes’ love letter and Niklas Emqvist’s clever animated gifs helped me realize I’m more emotionally connected to this community than I thought.

Reflecting on lessons learned

Being a part of the visualization community, and InfoVis in particular, has taught me a lot. On the one hand, I think I learned to take myself more seriously by seeing how researchers I’d long respected — John Stasko, Sheelagh Carpendale, Marti Hearst, Jeff Heer, to name just a few — took me seriously even when I was just starting out. I feel truly lucky to have stumbled into a community that embraces both rigor and new ideas.

Observing that I seemed to have a knack for writing papers that pissed certain people off also helped me realize that I like speaking up when I see a problem, even if I know what I’m proposing is contentious. This shouldn’t have come as a surprise (my life before I got into visualization and CS is full of extremism and risk-taking as I tried to figure out what problems mattered most to me!) But somehow it felt like a revelation.

On critique, and learning from failure

Lately I’ve been thinking a lot about criticism in research. Being publicly criticized early, and repeatedly, has certainly changed my career.

On the one hand, it made many experiences, like conferences or interactions with other visualization researchers, a lot less fun for awhile. At a time when I most wanted to be accepted and feel comfortable sharing my ideas, I felt like it was out of reach. It can be hard not to wonder how my career might have been different, or at least my metrics, if I hadn’t started off in the negative limelight (not to mention returned there for a paper that I really liked), with all of the impacts it had to my confidence. It made it harder at times to relate to people who as far as I could tell had much smoother rides.

At the same time, I feel somewhat indebted to those critiques, not necessarily for their specific content but for keeping me on my toes. I’ve always been pretty careful with what I choose to speak up about, but I undoubtedly got more careful and rigorous when my work was publicly scrutinized. For this I’m grateful. These experiences may have made me think more deeply than I otherwise would have about the various contradictions that characterize scientific communication, such as how bold claims are rewarded even when many of the effects we deal with are relatively small. That my research ended up converging on challenges and problematic incentives of uncertainty communication might not be a coincidence.

It can also be transformative to have your worst fears come true. There’s an undeniable “If you ain’t got nothing, you’ve got, nothing to lose” feeling that’s very liberating. As someone who has always cared deeply about my work, it’s probably been a good thing to have external forces show me the danger of being too attached to how any single piece of work is received. I got very good at finding solace alone in my work and following my instincts about what the important problems are. I see all of these lessons as good things. I’m honestly not sure I would have wanted to be a Ph.D. student who had worked only on safe, mainstream ideas. There’s so much to learn, personally and intellectually, from rejection and criticism.

My only regret may be that it took me so long to realize that I was accepted by the community. Over the years there have been multiple times when after participating in a VIS conference, or running into vis researchers at CHI or another event, I would feel newly surprised at the level of encouragement and open-mindedness. When you spend a lot of time in your head harshly critiquing yourself, it can be hard to accept that that’s not necessarily how everyone else sees you. Part of my reason for writing this is the possibility that it will encourage junior researchers in visualization or other subfields of CS who feel misunderstood or “outside the core” to have some hope, even when things seem to go as badly as possible. Lots can happen even in a short career!

My experiences have made me think carefully about the immense value public criticism and discussion of methods and problems can bring. This may seem strange coming from someone who describes above how hard it can be to be in the public eye. True, I may never write scathing blog posts calling the entire field pseudo-science, and I probably wouldn’t wish what happened to me on others, at least not the second time around. But I’m pretty convinced of the value of open discussion of flaws and limitations in the work we do, of what is and isn’t an important problem, and of admitting when we’re wrong.

I think we could go further as a community in this direction, and appreciate, rather than alienate, those who speak up to critique. After all, the truth is often both beautiful and ugly. We can learn a lot from things we don’t want to hear. As the visualization research community becomes more sophisticated in its awareness of statistical reform practices like pre-registration and estimation, and as we gradually study more nuanced phenomena because we’ve exhausted bigger and more obvious effects, I have some concerns about where we’ll end up. For instance, we will delude ourselves into thinking that if we just use the right practices, it will solve all of the challenges of doing rigorous, important work? Being openly critical about the limitations built-in to what we do is a way to avoid becoming complacent.

At the end of the day, I care deeply about visualization research, and want to see it continue to grow in depth, relevance to the world, and rigor. If any community can be both open-minded and brutally honest at the same time, I believe it’s the visualization research community!