Qualitative Illiteracy and the One-Eyed Business.
The Facebook Ethnography Kerfuffle.
At the center of this kerfuffle is an anthropologist, Daniel Miller, his ethnographic research with teenagers in a small town in the UK, and a press report on a blog post about his research that went viral.
What’s exciting about this story — leaving aside the business implications for Facebook for a moment — is that we get to observe the treatment of qualitative research in its moment in the spotlight.
It’s not pretty.
Much of the drama came from the manner in which it was reported, which certainly is worthy of some discussion. Most came to it with hyped expectations about Facebook’s imminent demise. But there is more to the story.
Namely, a widespread and fundamental lack of understanding as to how qualitative aids decision-making by giving access to insights unavailable to quantitative.
What exactly did he say that got everyone in a huff? I revisited the post, worried that I had, in a few twitter exchanges, gotten ahead of myself. In the last paragraph, after laying out four strategic observations, Miller writes:
It follows from this that a change in Facebook can arise, not from anything that happens within Facebook itself, but because of changes in the other media it is differentiated from. In my surveys at schools it is now Twitter, Instagram, WhatsApp and Snapchat that connect pupils with other young people. Snapchat connects the closest friends, WhatsApp the quite close friends, Twitter the wider friends, while Instagram can include strangers. By contrast, Facebook has become the place where people interact with older people, especially parents and the wider family, or even older siblings who have gone to university. To prevent overgrazing, Facebook has to feed off somewhere else. It has thereby evolved into a very different animal — not that anyone seems to have noticed.
That, for the record, is exactly the kind of category evolution one would suspect in a rapidly changing space. It had been documented elsewhere previously, and even validated by Facebook itself! But, somehow, this isn’t perceived as a finding. It’s dismissed as ‘anecdote’ and, later, a ‘throwaway comment.’
The response by the business analyst and ‘dataphiliac’ crowd was predictable, yet brutal. One thing that is clear is that there is a lack of awareness of what qualitative research actually is and how to understand it.
Few stopped to consider the implications of the findings, and even fewer resisted the temptation to mock what they perceived as an actual lack of methodology. Uncertainty, or a lack of confidence about these questions leaves organizations at a serious disadvantage.
A sample tweet: “Is this really all there is behind that ‘FB dead w/UK teens’ story — a few assertions from a study in one UK town?”
First of all, to hear a topline report of findings from deep ethnographic research by an anthropologist with solid credentials be called “a few assertions” is a bit laughable, if you understand qualitative at all.
But it highlights the bias towards quantitative and against qualitative understanding in the modern business world. A lot of traditional qualitative research is useless. But true qualitative that is sensitive to anthropological and psychological truths are as real as any numbers.
But this is nothing new. In 1983 Rohit Deshpande identified the limitations of the quantitative bias even within marketing in “Paradigms Lost: On Theory and Method in Research in Marketing”
Only recently have marketing scientists become concerned with issues in the philosophy of science. This paper points to one neglected area — the implications of a theoretical tradition for the selection of research methods (design, data collection, and data analysis). It is argued that marketing has been relying primarily on only one theoretical tradition. The dominance of this philosophy has led to marketing science growing more rapidly in the area of hypothesis testing than in the development of new, rich explanatory theories.
In 2006, Saleh Alshebil of University of Texas at Arlington, revisited Deshpande’s claims and performed a content analysis of marketing literature, “Are ‘Paradigms Lost’ in Marketing? Some Twenty Years Later…”
We analyzed a total of 394 articles... Almost half (47.2%) of all the articles published were of the quantitative methodology. Purely qualitative articles made up a mere 7.1% and articles containing both quantitative and qualitative research methodologies made up 32.7%.
It is not a new argument that organizations learn almost exclusively through quantitative analysis of the world. What seems to drop away, however, is the argument about how this bias against qualitative leaves organizations ill informed and vulnerable.
The most common argument against qualitative is sample size. Let’s address this, and revisit the role of qualitative research. Here’s what wikipedia says about it:
The qualitative method investigates the why and how of decision making, not just what, where, when. Hence, smaller but focused samples are more often used than large samples.
And, here is Margarete Sandelowski, in a much cited article, “The problem of qualitative rigor”:
Samples in qualitative research are often not representative in the quantitative sense, but in qualitative research, any subject belonging to a specified group is considered to represent that group. Anyone’s experience, if well described, represents a ‘slice from the life world,’ and is, therefore, appropriate subject matter for qualitative inquiry. The task of the qualitative researcher is to establish the position of all subjects in relation to the group of which they are members and the meaning of their slices of life.
And this, from “The logic of small samples in interview-based qualitative research.”
If, then, we accept that small-scale interview-based — and radically qualitative — research is intentionally conceptually generative, its findings…cannot escape the charge of speculation. Nor should they have to do so. It is in the nature of exploratory studies to indicate rather than conclude. More strictly put, such studies formulate propositions rather than set out to verify them — or, at least, convincingly demonstrate them (through reliance, for example, on ‘‘representativeness’’ and the persuasive weight of large samples).
Despite the hue and cry around this particular ethnography, elsewhere in the world, there are real signs of a new embrace for true qualitative methodologies.
While Daniel Miller wrote a defense-slash-mea culpa on the role of academic publishing and virality while on vacation, Netflix had recently hired Grant McCracken to share his findings on the phenomenon of “binge viewing.”
The question of qualitative vs quantitative is not a real question that people serious about understanding the market ask. The real question is, how can qualitative help quantitative make businesses smarter.
We need more Daniel Millers and Grant McCrackens, and more collaboration with the dataphiliacs.
I applaud Daniel Miller for his work. The debate about its validity represents a powerful opportunity for all of us who seek to transform business through empathy to champion what I have come to call brand listening.
It’s listening because it’s face-to-face. It is brand listening because it honors the deep anthropological significance brands play in our lives.
And it is strategic, because it’s in pursuit of the motivations in a category — the only thing that matters in a rapidly evolving marketplace. For in every category, it is the brand that is able to manifest the motivations of the category through product that remains.
As Rohit Deshpande wrote:
By excluding alternative methodologies, marketing scientists are perhaps unknowingly also constraining themselves into a set of only partially appropriate techniques for a limited subset of marketing problems. Only a few marketing scholars have recognized this fact….Moreover, the methodologies that have been developed and tested in marketing research are increasingly those suited to confirming propositions or hypotheses rather than to discovering new propositions or hypotheses.
These past two days have demonstrated that to perfection. While the implications of Daniel Miller’s work never really got discussed by the “real” businesspeople for lack of data, the data appears to validate him.
Maybe Facebook had already learned this through their own ‘social listening.’ Maybe this wasn’t news to them. But the startling and brutal fact of our dynamic marketplace is that change is the only constant, and the need to remain in contact with, and sensitive to the shifts in experience is invaluable.
For this is the true purpose of qualitative: to explore the experience of a phenomenon from the perspective of the person experiencing it, in order to develop hypotheses for quant to explore. For qual is expert at what quant is not: illuminating the nature of constantly shifting relationships, not the fact of these relationships.
The only real risk is in not listening.
For, it might just become a collection of anecdotes, gathered and interpreted by a social scientist, that is shared when people talk about how and why many young people began to limit Facebook’s access to parts of their lives.
I’m sure Facebook is.
Without our qualitative muscles, we go out into the world as a one-eyed organization, learning only through confirmation and validation, closing the door to unseen opportunities.
I imagine in a few short months there will be someone who shows up with a quant friendly report that indicates that the nature of Facebook usage shifts around this age and, while they still use it, it is not as central to their identity as it once was.
Even more likely we begin to notice Facebook doing things a bit differently.
I awoke this morning to a story in Forbes that read:
For Facebook, the trend was clear: User growth is fastest in the oldest cohort.
Huh. That sounds familiar. Where had I heard that before?
Many thanks to Grant McCracken for pointing me towards Rohit Deshpande’s piece many months ago.
Additional reading: Please check out Tricia Wang’s beautiful “The Conceit of Oracles: How we ended of up in a world where quantitative data is more valued than qualitative data.”