How do we establish Truth?
(Part 2)
The problem I brought up in the first part of this article can be boiled down to the reality of unknown unknowns. There are just too many things that we just can’t know ahead of time, which is why we often have to go out and find new information. So the question then becomes ‘how do we go about getting new information?’
And part of the answer lies in the scientific method and statistical truth.
The scientific method is an extremely powerful tool and it’s a blessing that human beings discovered it. Because before that people were subject to all kinds of biased ways of thinking. Superstition, witchcraft, magic, snake oil, religion, human sacrifice, holy wars. These were all products of imagination not grounded in science.
And the scientific method isn’t proposing anything that complicated. It all starts with some kind of observation or idea, and that any idea, or hypothesis, should be tested rigorously before it is taken as fact.
This the hypothesis should be stated in a way that it can be tested. It means external variables should be controlled for. Any testing should be repeatable. Biases, both cognitive and otherwise, should be eliminated or taken into account. Alternative explanations should be examined. And a number of other relatively common sense notions.
Now that’s not to say that without science we were incapable of having good ideas, or insights into the nature of reality. Our brains are powerful pattern-matching tools and we can learn a lot through experience and trial and error.
A lot of churches do charity work because they understand the value of social cohesion. Some homeopathic remedies actually are found to have healing properties, which makes sense since a lot of clinically tested drugs originated from plant extracts. And some things, like magnetism, even though they are well understood scientifically, still have something almost magical about them, which is why they capture our imagination.
But Goya had a famous quote: ‘Imagination abandoned by reason produces impossible monsters. United with reason, it is the mother of the arts and origin of their marvels.’ Science is about making sure we are learning the right lessons, and not imagining the wrong ones.
Because at it’s heart, science is all about telling a story. A story that does a good job of modeling reality, and that allows people to make correct predictions consistently. Because a good story will let you see objects and events billions of miles away, nanometers in length, or years into the future or past.
But coming up with the right story requires a certain amount of disciplined thinking, which is why science is intimately tied to the use of statistics.
If you are repeating an experiment over and over again, you will end up with multiple measurements that you can examine in aggregate. Sometimes there are minor measurement or human errors. There might be variables that are unaccounted for. There could be any number of reasons for measurements of the same event to give slightly different results.
Statistics allows scientists to tease out a signal from what otherwise would seem like random noise. To find real patterns in apparent randomness.
And one way to do that is through the ‘p-value.’
What the p-value represents is the ‘certainty’ with which a signal is in fact a signal. So a p-value of 0.05 says there is a 5% (or 1 in 20) chance that the pattern I’m looking at is due to randomness. The measurements might have been off, and the pattern just happened to occur.
Now a p-value of 5% for most fields is taken to be ‘significant.’ Noteworthy. Or in other words, worthy of publishing. And there are some problems with this. For one, if 20 scientists perform the same experiment, you would expect one to get ‘significant’ results just by chance. And in the scientific community, positive results are published more often than negative results. This skews the perception of how significant results really are. It’s one part institutional failure and one part cognitive bias.
Derek from Veritasium does a good job of explaining those problems here.
Part of the problem is related to something called Baye’s Rule. I want to do a whole post about that, but for now you can think of it like this: Suppose we do some kind of test and we get a negative result for that test. What are the odds that the negative result was correct?
Doctors have to deal with this all the time with false positives or negatives in medical tests. What are the odds that what I’m looking at is true, given that my measurement said it’s true? And conversely, what are the odds that it’s false, given that my measurement said it’s true?
Because all we know are the results we see, and from those we have to infer how likely those results are to be True.
This randomness is at the heart of the problem with the scientific method. Even if we fix bureaucracy of modern institutions and publications, and the culture of ‘publish or perish’. Even if we could eliminate statistical, measurement, experimental or human error. Even if we could fix all of our own cognitive biases, there comes a point where the results of a study need to be either accepted or rejected.
And there isn’t a well-defined rule for when you should do this. Some fields need a p-value of 5%, others of 0.000001%. And sure you can be more and more confident with smaller and smaller p-values, and you can have more people do the experiment, or build better measurement tools.
But at some point you have to take a leap of faith and trust the results of the process. And this can be scary.
Now, this may not be true for simple things. If I were to throw a ball and make some predictions about it’s speed, acceleration and distance traveled using the equations of motion, most people could be convinced of the predictive powers of the model. However, this could partially be explained by the model coinciding with their daily experience.
If you get into something big, complicated, and not easily summarized in day-to-day life, like global warming or even evolution, a lot people don’t seem to believe it. Even though there are thousands of scientists, mountains of data and complex mathematical models all pointing to the same thing, there are still millions of people who just don’t believe it. To them it’s all a conspiracy and anything you add to it just makes the conspiracy bigger.
‘Those people are in on it too.’
Part of this is because they don’t have the background the understand the science. Part of it is political bias and punditry. And part of it is propaganda by people with a personal stake in collective ignorance.
But fundamentally, it all comes down to trust.
Which brings me to my third and final part.
