Data Visualization Feedback: Evaluating Choices, Not Checklists

Joshua Smith

Follow

Jun 13, 2019 · 7 min read

I’ve encountered a scenario often enough that I’m willing to guess you have, too. You put together a data visualization piece, toss it out to the community, and the feedback starts rolling in (perhaps layered into “praise sandwiches”):

“Don’t use red, people connect it to something negative.”

“Don’t reverse your y-axis, it’s confusing”

“You need more white space, it feels cluttered”.

Directives (“do this” or “don’t do that”) flood in, leaving you with a task list of changes to make. The community encourages us to iterate on feedback, so we offer up a version 2 for more criticism, and then perhaps a version 3, and so on — until we’re left with something we call “done” after crossing off all of the to-do’s we were given.

Directive feedback usually comes from a helpful place, often well-grounded in data visualization theory and best practices. However, directives fail to take into account the creator’s design strategy: why did the creator choose to do things a certain way? Our data visualization works deliver not just information and insights, but an experience. That experience might be purely informative, but many times we’re designing for something more.

Take, for example, Simon Scarr’s devastating Iraq’s bloody toll about deaths in the Iraq war.

Imagine if, in an earlier version, Simon Scarr had been told not to reverse the y-axis, or to avoid the color red? Modifying these two elements completely alters the experience, as Andy Cotgreave showed in his redesign.

I worry if our directive feedback, ignorant of the design strategy behind the creator’s choices, may be diverging works away from the intended experience.

I’ve taken a number of creative arts workshops, mostly focused on writing. Creative workshops, most commonly, are a collection of artists / authors that provide each other feedback in a peer-to-peer forum, usually mediated by a more experienced artist (often a teaching professor). In these workshops I heard a consistent refrain from every workshop leader: “Know the rules so you can break them.” In the creative arts, the real magic comes when you know the rules well enough that you can meaningfully deviate from them. Each broken rule delivers an experience that wouldn’t have been possible by following the rule.

For example, take the bizarre language of Buffalo Bill ’s by e.e. cummings.

In this poem, Cummings slaughtered English grammar enough that you’re forced to read each line and each word in a completely different frame of reference than typical syntax would allow. The experience is very different than that of Shakespeare (who also broke rules, but different ones) or Wordsworth (who…also…broke rules).

Consequently, providing feedback in the creative workshops was much more than grammar checklists. Because we were encouraged to explore boundaries and break rules, our feedback was less focused on “best practices” and more on the overall success of a piece. I would highlight 3 unique elements of feedback in creative workshops:

1. We described our experience of engaging with the piece. When I would receive my marked-up poems back, lines and words would be circled and underlined, with arrows to margin comments like “this feels important”, or “I can really see this image”, or “I sense a strong shift in tone here”. This kind of feedback helped me understand the experience I was delivering (intended or not), and how my choices created that experience. This, in turn, allowed me to revise and shape my works to refine the experience I’d hoped to deliver.
2. The “volume” metaphor isolated elements that contributed to the experience. A really common phrase I heard was “this starts to <insert effect>, but I’d love to see you turn the volume up on that.” This was a way for us to tell one another that something was starting to work, but it needed more of an impact. This is especially pertinent for rule-breaking: a minor deviation from best practices may only signal a mistake, whereas stronger deviations signal meaning. For example, William Faulkner is known for run-on sentences which create a very unique cadence. But, suppose he had fewer run-on sentences? Or shorter run-ons? It might have simply suggested he’d written with poor grammar. If such a piece had been in one of my writing workshops, I and others would probably have said “your run-on sentences create an interesting and unique cadence that really fits your themes, but I’d love to see you turn the volume up on that”.
3. Directives were replaced with (and received as) ideas. I’m not going to say that my poems never had editorial mark-ups. But there was a much different culture, where these mark-ups, within the context of numbers 1 and 2 above, were seen as suggestions to refine the experience. My poem might have mark-ups for new line breaks, but when paired with feedback such as “your tone is creating a sense of anticipation and fear”, I can now see the suggestion as an idea to emphasize that tone. In this sense, any “directives” I might receive felt more analogous to brainstorming.

In a sense, the feedback from creative writing workshops were different because they were all about identifying, and then emphasizing, the intended experience — not just the literal meaning of the words. In the same sense, good data visualization feedback identifies and emphasizes the user experience, not just the literal interpretation of the data.

Rather than directive feedback, which ignore the creator’s design strategy, the kind of feedback found in creative workshops help the creator understand how each choice executes that strategy, whether successful or unsuccessful. This kind of feedback not only welcomes more innovative pieces, but also serves as part of the act of creation. I’ve run a number of creative workshops for data visualization, and the transformation between a version 1 and a version 2 is dramatically different than the differences from editorial markups. Creative pieces are often completely re-imagined into something even more powerful.

For example, see this version 1 piece submitted by Katelyn Schaub for a creative data visualization workshop I hosted:

After the workshop provided feedback in a format similar to creative arts workshops, here is her version 2, which was awarded Viz of the Day by Tableau Public:

The transformation is powerful. The charts changed and new data was added, but more importantly the experience changed. The tone and style create a completely different emotional engagement than the original piece.

Both Ken Flerlage, Tableau Zen Master, and I have written about writing as an analogy for data visualization, specifically in reference to rule-breaking (you can find the pieces here and here). Much like in other creative arts, the choices we make accomplish more than simply the literal interpretation of the content — they create an experience.

Creative workshops, then, can serve as a strong paradigm for how we provide feedback on data visualization, including the art behind breaking rules. To do this, we need to focus less on simple checklists of best practices, and more on how the creator’s choices deliver an experience. We’d do well to bring in the practices I’d seen above:

1. Describe your experience: How did you feel when you engaged with this piece? Were any emotions stirred? What did you do with it — did you just read, did you skim certain sections, did you click on things or hover your mouse in certain places? What did you expect to happen, that didn’t? And what happened that you didn’t expect?
2. Use the “volume” metaphor. Identify the elements in the design that contribute to your experience. Do these need turned up? Perhaps turned down? Are there broken rules that just feel like broken rules, instead of meaningful choices? Are there particular choices that you think simply detract or confound from the experience?
3. Offer ideas. Once you’ve taken the time to do numbers 1 and 2, you might offer ideas on how to “turn the volume up/down”. The context makes these ideas feel less like directives, but some considerate phrasing can help the delivery: “Have you considered…?”, or “I might try…”, or even “I would suggest…”. This kind of feedback invites the creator to return back to their design strategy, and evaluate whether or not your ideas might better deliver the desired experience.

Providing feedback is an opportunity to participate in the act of creation by showing the creator their work from a different angle. Directive feedback based on fundamental checklists fails to help the creator compare your experience to the intended experience. When we’re asked to provide feedback, we’re asked to help shape an experience.