Anecdotes Aren’t Data
However truthful anecdotes are, they do not represent statistical evidence
An anecdote is a short, informal account about something. Anecdotal evidence is a collection of anecdotes. A defining feature of anecdotes is that they consist of real incidents experienced by real individuals. That said, anecdotes do not typically represent a common experience, and thus do not break or make false the applicable generality the anecdotes refer to. In other words, exceptions do not invalidate the rule. And yet anecdotal “evidence” is very commonly (and mistakenly) used to attempt proving that exceptions are the rule.
A fundamental problem is that despite however truthful anecdotes are, they do not represent statistical evidence, the standard for assessing overall generalities and their exceptions. The raison d’être of anecdotes is to describe individual experience, not to demonstrate the frequency of the experience. Nonetheless, anecdotes are commonly used to assert a larger point of view as to the shortcomings and failings of policies, laws and regulations.
Someone I know recently insisted he knew ten people who, as a result of Obamacare, were paying significantly higher premiums and deductibles for their health insurance. Given the statistically estimated numbers of those in such a circumstance, his assertion of ten individuals seems dubious. He actually only mentioned one individual — who he had cited at least two previous times. When I noted that some 15 million people now had health care coverage who couldn’t afford it previously, he replied the law shouldn’t have been passed without all exceptions being accounted for. I pointed out that large, complex legislation always has issues that need to be addressed, but given the dysfunctional political realities on one side of the aisle, such tweaking was not politically possible. He then added an anecdote about the health care of someone he knew who was from Canada, “proving” why national health care was fundamentally flawed.
To be clear, anecdotes such as these turn out to be largely untrue and/or are missing significant information. In fact, the accuracy and completeness of anecdotes in general are unknown. Yet this person had written off millions of individuals and families now with health care coverage simply because a much smaller number were paying more. What is missing from this is that the implementation of a vaguely national health care scheme that uses insurance companies is going to have issues not found in real single-payer systems. Then there is the economic reality that unless the pool of coverage includes large numbers of younger healthy people, and particularly so with a profit-driven insurance industry in the middle, costs for some will have to be higher. By definition, true single-payer systems are government funded and do not require insurance companies.
There are few issues that don’t have an anecdotal component. But these informal accounts don’t tell us much. Imperfections, shortcomings and trade-offs are all part of human endeavors. Unintended consequences are very high on the list. A few years ago I wrote about unintended consequences, and noted that as helmet laws for bicyclists were implemented, the numbers of serious injuries occurring did not decline as much as they should have statistically. It turned out, upon investigations by researchers, that some riders were taking greater risks because they felt safer with a helmet on. The anecdotal evidence might have indicated that helmets were not helping, but that wasn’t the issue at all. Human nature was.
Pragmatically, I consider anecdotes an artifact of complexity and variability. The phrases “I know someone…” or “I read about someone…” do not constitute viable assessments of issues. Data — statistics, percentages, trends — tell us what the greater truth is. Anecdotes tell us what the exceptions to that truth are, but without statistical scale they have no context. For this reason and others, the integrity of data is critical to its value. Not only must modeling be valid and data acquisition accurate, but statistical analysis must conform to recognized standards. Selective use of data to create a desired result violates these standards, as does starting with assumptions that data doesn’t support.
Life as we know it operates on vast amounts of data. The variety of data, the volume of it and the use of algorithms to search for patterns and correlations within it have revealed multitudes of insights regarding our planet, our societies, our economics, our behaviors and our results. Anecdotal evidence simply cannot provide this kind and quality of information. While it might reveal personal experience, it doesn’t tell us enough to know if it’s an anomaly or a pattern, or how significant it is. That’s what data are for.
The difference between anecdote and data is how easily the former distorts perceptions and causes some to form inaccurate opinions. When I was working on my graduate studies in history, the use of data was fairly new to the field, but it was already demonstrating that who, what, where, when and why made a lot more sense when there was data mining included. One study in my field of urban history used data to demonstrate how little socio-economic mobility actually existed among generations in the American colonies. Now big data is a vital part of historiography.
Anecdotes appeal to the emotional, intuitive parts of our brain, but this needs to be balanced with the intellectual, rational parts so we can use logic to place anecdotal information into perspective. The applicability of anecdotal evidence is fairly marginal without accompanying data. This isn’t being elitist — it’s functional pragmatism. One can be sympathetic as seems reasonable about anecdotes, but extrapolating these short, informal stories into anything more than isolated examples is going to lead to suppositions without supporting data instead of justified conclusions with it.
