# Uncertainty, “Right Answers,” and Conventions

I’ve written a version of this post before. Perhaps I’ll be repeating a lot of things that I have said earlier, although the argument will be extended slightly further.

The problem with the data is that it does not usually tell us what the right answer is. Rather, it tells us which answers are more likely than others — it’s the whole idea behind statistics: if we knew the right answers, we would need neither data nor their analysis. The “powerful” tests (and data), in the statistical sense, are those where we can distinguish between more right and less right answers more readily, partly because we have enough data in terms of sample size and also because the possible answers are sufficiently different from one another that most of them are highly improbable given the data on hand. Typically, however, finding powerful tests (perhaps defined more broadly than the usual sense) requires some amount of “cheating,” or ruling out possible answers on some conjecture that is not necessarily supported by the data on hand — call them theory, priors, assumptions, or worst of all, “logic” or “rationality.” All are justifiable, but only with the caveat that they limit the applicability of the answer beyond the immediate problem, with some special attention paid to the definition of what the “immediate” problem signifies.

High variance distributions, in this sense, are extremely valuable things, even if they may not allow for a powerful test that neatly separates the “more right answer” from the “less right answers.” That some questions don’t have neatly defined right answers, that there are many potential answers that are compatible with the data on hand, and that some problems really don’t have a clear answer — unless you cheat and change the problem appropriately, by imposing conditionalities by assumption, theory, etc. — is a valuable insight. In the same vein, if you do find a neat “right” answer, and did so by “cheating,” the important thing is to understand how you cheated to get to the answer, and whether you can repeat the same cheat when the circumstances have changed. Understanding where the distributions are fat may not be “useful,” but is certainly insightful — provided that you are not charged with finding the “best answer” by tomorrow.

An interesting social phenomenon, in this regard, is that is “conventions,” although many other words, such as “stories,” “beliefs,” “cultures,” etc. might fit in the same mold. Nobody has a convention that says 1+1 = 3: it is too obviously wrong an answer most of the time. Perhaps there are times where things might be mistaken for affirmation that 1+1 = 3 might be true…but they are too obviously unnatural and contrived that nobody would take them to be natural. But on all matters that are more complex, ideas that may or may not be factually “true” are routinely taken as “conventional,” ranging from creationism (generally factually wrong to varying degrees…depending on what you mean by “creation.”) to national myths (usually factually untrue, but facts are usually irrelevant) to ideological beliefs (cannot be factually wrong by definition) to textbooks (which may be “true” factually, but they are rarely questioned or tested by the students who use them).

The premise is that there is no immediately obviously right answer so pretty much any answer is acceptable — or, at least, not ruled out readily as obviously “less right” conditional on the data. One seemingly “rational” solution is to accept any answer that is not clearly “far less right,” but, far more frequently, people come up with the “tribal” answer that demonstrates the membership of the right tribe, shared experiences of the same group, adherence to the same “textbooks,” etc and enforce it as “the right answer,” at least among some circles. The tribal answer cannot be ruled out since it is not (demonstrably) wrong — why 1+1 = 3 usually does not qualify as a potential tribal answer. But, in such instances, being “factually right” is joined by being “morally right” (whatever that really means — but certainly specific conditional on a particular group) as the criterion for being “the right answer.”

Say, I might say Pi = 25/8 while others say Pi = 22/7. In this example, is 25/8 wrong? Well, somewhat — it is “more wrong” than 22/7, in the sense that |25/8 — pi| > |22/7 — pi|, but given that actual pi is neither 22/7 or 25/8, 25/8 is not exactly categorically “wrong.” Indeed, for all we know, someone who insists on using 25/8 as the value for Pi might do so for some reason that may actually be mathematically legitimate — he/she might have derived this value him/herself — which would be much more laudable than someone who uses 22/7 because “the magic formula/tribal elder/myth says so.” Often, the magic formula is not explicit: we believe certain stories/myths/conventions because that’s simply the way the answer had “always” been presented. A World War II German POW camp guard might believe (based on true stories) that the man who went by in the darkness who was wearing something that looks vaguely like a German officer’s uniform is in fact an officer, notwithstanding the lack of data. An escaped POW during World War II, indeed, described the problem of escaping from POW camps as fundamentally a psychological problem: “reliance on inattentiveness of mankind, the willingness of people to accept the everyday events on their face.” Of course, this can be easily recast as a statistical problem: given a small sample of data drawn from a specific situation, we haven’t got enough observations to reach a conclusion with any conviction. “We” do (or we think we do, whoever this “we” is), however, have a lot of data from “situations like this” that went into forming the conventions that define us, and besides, what’s the consequence of one erroneous identification anyways? Since we don’t know better, why not just take the convention for what it is rather than make needless trouble? (especially since mistaking an actual officer for escaping POW is likely to land the guard in hot water anyways.)

This is, in a sense, a perfectly rational and reasonable disposition. This is also how societies stagnate and how seeming reliance on empiricism can actually reinforce obscurantism and stifle innovation. A lot of data is generated by the “everyday events.” Conventions are built on everyday events. The potentially illuminating data come from odd and unusual events — it’s not every day that an Allied POW escapes from a German POW camp disguised as a German officer. Odd events rarely supply enough observations to provide empirically convincing “right” answers. It is easy, when the “right answer” cannot be easily determined empirically, to pretend that we know the “right answer,” more than we can justify on data alone, because we know formulas, myths, stories, and so forth. This is where things can get potentially dangerous, I think.