How Scientific Breakthroughs Work
I have a good friend and colleague who is a very accomplished engineering leader. During lunch yesterday, he told me a story about how he solved what at first seemed to be an extraordinarily challenging problem.
The first instances of a new computer processing chip that his team had designed had returned from the factory, and it didn’t seem to be functioning in the way that everyone expected. Immediately, people started to hypothesize about the cause, and, based on very little of the existing data, a consensus had been reached that it was Cause A, which represented a fundamental flaw in the architecture of the chip. Almost everyone involved was convinced that Cause A was the problem. A ton of effort was then deployed to try to find a work-around for Cause A.
Instead of jumping on the bandwagon, my friend examined the overall characteristics of how the problem was presenting, by getting a big picture of the available data. He noticed that there was a systematic pattern to the incorrect functionality, and that in fact you could look at the functionality from a different perspective and it could be perceived as not being broken at all. His thinking was that if Cause A really was the problem, then the observed behavior should have not only been completely wrong, but randomly wrong.
All scientific breakthroughs occur when commonly held beliefs are set aside, and when the available data is examined with fresh eyes and a willing heart.
My friend tried to alert others to this inconsistency, but Cause A had gained so much traction that his voice was lost in the cacophony. Instead of giving up, my friend, the leader of a large organization, decided to personally dig deeper into the data. He downloaded so much data onto his laptop that it almost filled the hard disk. He then started poring over this data, line by line, trying to make sense of it.
After a day of this, he could see what was happening. The problem was not related to Cause A at all. In fact, the chip was doing the right thing, but instead the mechanism that was automatically interpreting the output of the chip was not functioning correctly. It turned out that the real cause, Cause B, was completely different from the incorrectly hypothesized Cause A. It ended up being very easy to fix Cause B: a few lines of code needed to be changed in the test software.
My friend’s story is not unusual in the world of science and engineering. This kind of scenario occurs regularly in all organizations and institutions. To me, it seemed like a perfect demonstration of concepts that I’ve been pondering regarding scientific breakthroughs. All scientific breakthroughs occur when commonly held beliefs are set aside, and when the available data is examined with fresh eyes and a willing heart.
In this vein, a good example is Albert Einstein’s discovery of the special theory of relatively: the speed of light in a vacuum is the same for all non-accelerating observers (regardless of their relative velocities), and therefore the sequence of spatially-separated, non-causal events is dependent on the inertial reference frame of the observer relative to the events. The Lorentz transformation is, in fact, the more universally-true, data-reality-congruent way of moving between inertial reference frames (not the Galilean transformation).
Einstein was disturbed by the fact that, while Maxwell’s equations seemed to very accurately describe electromagnetic energy (traveling at the speed of light), they were not compatible with the classical mechanics as developed by Newton, which seemed to accurately describe matter (when traveling at relatively low velocity).
It’s not that I’m so smart, it’s just that I stay with problems longer. — Albert Einstein
Instead of buying-into the consensus agreement that there was something not-quite-right with Maxwell’s equations—after all, the data showed that they were, in fact, correct—he set aside all prejudice and instead began to run thought experiments that involved considering a person traveling in a train past a station at close to the speed of light.
Einstein set about debugging the actual problem, by diving into the junction where the two theories broke down. Keeping his mind open, and assuming that the data was correct, enabled him to actually understand what was going on.
By doing this, he was able to determine that we had been making an incorrect assumption about classical mechanics: that the correct way to move between inertial reference frames is by using the Galilean transformation, which is approximately correct at the low relative velocities that we experience in everyday life. By removing that assumption, and instead using the Lorentz transformation, he was able to reconcile classical mechanics with Maxwell’s equations, producing a consistent model of reality. This improved model has now been thoroughly confirmed by additional data from further experiments.
The important thing is to not stop questioning. Curiosity has its own reason for existing.—Albert Einstein
Humans seem to have a built-in need to be in control, to know, to be certain, and to be right. We like to think we have all the answers. We learn a few things and then we think we know everything about how reality works. We assert our knowledge, even in the face of data that contradicts it. The “science says” crowd is no more free from this affliction than the “scripture says” crowd. I personally get more concerned by “science says,” because to me it’s a corruption of the spirit of science. To me, it’s an abomination.
The spirit of science is not: “You’re wrong, Einstein! Maxwell’s equations are clearly wrong. Why are you such a pseudoscientist to assume that that they’re right? We know they’re wrong: Maxwell’s equations are not invariant under Galilean transformations, stupid!”
The true spirit of science is this: “These are our best models of reality. Here is where the data matches the models, and here is where it doesn’t. Here are the things that don’t make sense yet. Here are the opportunities to explore more deeply. Let’s open our minds to all possibilities to see if we can find a theory that explains this. Then, once we have a theory, let’s test it against reality.”
By the way, I suspect that this approach can be used very effectively in all areas of our lives.