What does data look like to you?
Metrics often seem inherently factual, but they have shortcomings too. So we shouldn’t throw out data that isn’t numeric, or dress it up in numbers to seem more credible. This distorts the truth and reinforces the fallacy that “real” data must be quantitative.
Data comes in diverse forms. On the Atlassian Research & Insights team, we embrace a broader definition of data — one that includes both quantity and quality — to help us better understand our customers and make more informed decisions.
When you think of the word “data,” what do you imagine?
Maybe something like this…
But probably not this…
Our default association with data is often quantitative, aggregated by volume or time. A 4.5-star Yelp rating somehow seems more factual and definitive than the aggregated pages of detailed, in-depth text reviews. Sure, a numeric rating is easier to immediately grasp — and our increasingly short attention spans call for speedy understanding. But both the rating and the reviews are valid forms of data.
The appeal of numbers…
Abstract ideals, like “customer delight” or “excellent performance” are not very actionable. Enter surrogation: the subconscious tendency to substitute metrics for meaning.
“Whenever metrics are present, people tend to surrogate,” writes Michael Harris and Bill Tayler in the Harvard Business Review. Nobel prize winner Daniel Kahneman and Yale professor Shane Frederick note that common conditions of surrogation include when the objective or strategy is fairly abstract but the metric of the strategy is concrete and conspicuous.
The desire to track metrics is a logical byproduct of wanting to make something nebulous a bit more tangible. But is my 5-star rating the same as yours? As Erika Hall writes in Just Enough Research, “There is no way to tell what the mean means.” Despite their apparent objectivity, numeric data can be subjective too — particularly when used to describe user experiences.
…and their danger.
A bias to revere metrics (and ignore all other observations) can become dangerous when it impacts our decision making. The mistaken belief that numbers are our only compass has affected many throughout history, including Robert McNamara, U.S. Secretary of Defense during the Vietnam War. For McNamara, war was reduced to simple math: make sure enemy deaths > one’s own. But McNamara’s belief led to defeat — not to mention tremendous loss of life.
Companies also risk missing important phenomena when we disregard data that don’t come from or fit into numeric models. In her TED Talk, technology ethnographer Tricia Wang highlights the example of Nokia: its refusal of her qualitative research findings — that poor residents in China were saving huge portions of their income to buy smartphones — led to a catastrophic loss of market share (indicated by the red line in the graph below).
Metrics can fail to capture the dynamism of human behavior, which is complex and constantly changing. We cannot fixate on numbers alone.
How many vs. For who
Numeric data is often about how common a phenomenon is: i.e. 80% do this, while 20% do Y. This is appealing for seeming to help us prioritize in a very utilitarian way. After all, shouldn’t we seek the greatest good for the greatest number? Too often this philosophical stance is implicitly embedded into user experience decisions, e.g. “If it impacts a large population, it must be important!” or “Look at how big that sample size is!” We fetishize scale despite not being very good at understanding very small and large magnitudes.
But in the context of business, some customers are more valuable than others. At Atlassian, large customers make up a minority of our customers, yet contribute more than half of our revenue. Despite being under-represented in population size, certain perspectives can have additional influence — whether it be an enterprise vs. a small or medium-sized business, an admin vs. an end-user, etc.
Numbers → numbing
Outside the realm of industry, focusing on numbers can also diminish the humanity behind them. “Psychic numbing” can be a natural survival tactic: for instance, coping with all of the COVID-19 deaths is much easier when reduced to a number — rather than the lost lives of hundreds of thousands of people.
The coronavirus is magnifying disproportionate impacts in many other aspects of life, including education and the economy, to name a few. Focusing on a numeric majority can help create a sense of progress in difficult circumstances, and yet also exacerbate inequality for those at the margins. Consider the case of remote learning, and how the students who aren’t connecting virtually probably need school the most. Tamasha Emedi, an assistant principal at an elementary school, explains in an interview with Every Little Thing:
“I think I just saw a survey that said 94% of kids have internet. What about those [other] 6% of kids?… We can’t be like, ‘Done!’ The majority is not the goal. I’m not going to be proud that 94% of families know what’s going on right now. I want 100%…The same people continually fall through the cracks of our system because we stop and celebrate once we get to 80 or 90%.”
Interlude: I’m not a hater
At this point, I should clarify that I’m not against numbers. Both quantitative and qualitative data have their uses, and they can be particularly powerful in tandem. As Sam Ladner writes in Mixed Methods (citing Creswell et al), quantitative research is useful for measuring the pervasiveness of things we already know, and qualitative research uncovers things we don’t know much about.
What I am against is (1) numbers as the default — and often, only — understanding of what data is and (2) prioritization by population size, at the expense of “marginal” populations.
Representation matters
There’s clearly a bias in our implicit understanding of what data is, and this is in part due to representation. Much of our data is biased because of those who are left out or underrepresented in datasets — such as women, people of color, and people with disabilities. These gaps have been highlighted by many including Caroline Criado Pérez, W.E.B. du Bois, and the AI Now Institute. Journalist Mona Chalabi also reveals that sometimes data have been collected but are hidden or misrepresented.
While we should be skeptical of charts and graphs, they often command an authority of truth — especially if they are computer-generated. It is thus tempting to represent qualitative data in “quant-y” ways too, given that charts and graphs are a familiar road. But when we expect all data to be represented in quantitative formats, we implicitly elevate the idea that numeracy = legitimacy. Qualitative data is neither inherently numeric nor meant to convey representativeness, so it shouldn’t conform to these modes of representation.
For instance, below are two different ways of representing the same data about how well customers perform certain tasks on Atlassian websites — with success indicated by green, partial success with yellow, and failure with red:
You’re entitled to your own opinion as to which of these is more effective at communicating the data. If we took a poll, I imagine there wouldn’t be a clear “winner” among the two. Even in broader circles, designers and statisticians disagree on what makes a data visualization “good”. But I’d argue that data from 10 research participants shouldn’t be represented quantitatively, as it is on the right. So how should we represent qualitative data?
Seeing is believing
Quotations and video clips can be vivid and accessible forms of media, but also potentially aren’t resonant or too time-consuming for some. Charts and diagrams are powerful and appealing precisely because they summarize information at a glance, and it’s possible to create visualizations with qualitative data. Giorgia Lupi and Stephanie Posavec are two designers who showcase how this can be done in novel yet rigorous ways.
These unconventional visualizations may not be immediately legible, but that’s part of the point. Piquing curiosity can encourage engagement and even be more memorable. The other aspect of this project that is notable is that form follows function. The hand-drawn style reinforces individual and human idiosyncrasies, not stripping each data point of its uniqueness with a more standardized, computer-generated graphic style. How you communicate mirrors what is communicated.
Returning to the same study about how well customers perform certain tasks on Atlassian websites, consider these experiments in representing the same data about experience journeys:
While the majority of people succeeded on task E, being successful required interacting with page elements and/or looping between multiple pages. Successful journeys were not always smooth ones. By contrast, people who successfully completed task G tended to so within one or two link clicks. Otherwise, people were misled to incorrect information, leading to failure on the task.
A far cry from Lupi & Posavec’s artistry, but I still had fun creating them. Moreover, exploring alternative forms of data representation raises questions not only about how to represent data, but what our understanding of data is. Looping repeatedly between the same set of pages, for instance, doesn’t need to be a formal metric to tell us that a user experience isn’t ideal.
Reconsider what “data” is and can be
A transaction. A duration of time. A facial expression. These are all forms of data, regardless of if they are quantified are not.
Whether it’s in the personal or professional realm, let’s think twice about what “counts” as data. One way to go about this is to try collecting and reflecting on your own data. Our phones collect information and subtly suggest what health metrics we should care about — for instance, step count, screen time, headphone decibel levels, etc. But what measures of health actually matter to you? Can the quality of your health be better described by other “variables” — deep sleep, hearty laughs, time outdoors, etc?
When it comes to our customers’ experiences, these too can be described with diverse forms of data. On the Atlassian Research & Insights team, we approach this with not only regular and methodical measurement — using the Happiness Tracking Survey (HaTS) — but also with product research and the synthesis of feedback from across the Atlassian ecosystem (including our online community, Reddit, Twitter, Medium, etc.) Basing our understanding on one form of data is like looking at only one color in a photograph, preventing us from seeing the whole picture vividly.
When you think of the word “data,” what do you imagine now?
Maybe your first association is still numeric, and that’s okay. Certainly, it’s simpler and faster to focus on one form of data. But embracing a broader definition of data enables us to have a richer, more nuanced understanding — both of our customers and ourselves.
