Calm Down — Machine Learning Isn’t Making Us Dumber
Lack Of Discipline & Understanding Is… And That Is Nothing New
Never let a good crisis go to waste. I am pretty sure some genius said that… smirk.
So now, at least according to the futurist and Rice University statistician Genevera Allen, machine learning has our science in crisis. Allen calls it a reproducibility crisis. The red squiggly line that appears when I type this reminds me that re-branding is once again in play.
Reproducibility along with a myriad of other jargon — repeatability, replicability, and other spell check annoying terms are just aspects of testing and hypothesis validation. Can someone else do what you did? And get a consistent result? In other words — did you really discover something or was it just luck. It all falls under peer review and none of it is new.
Nature called out the issue a few years ago. Hilda Bastian called it out in her blog tracing it back as far as 1964. Sadly, there is nothing new here. Peer review has been broken for a very long time.
In fairness, Machine Learning has been around just as long. So maybe it is the culprit? Of course, it wasn’t called machine learning back then. Or maybe this can all be summed up a slightly different and more informative way:
As long as there has been technology to quickly produce answers, there have been a lot of people happy to use it without having any clue if it is right.
Or if you prefer:
Calculators, Excel Formulas, and Pre-canned ML Functions are fast and easy. Statistical validation, logical discipline, and real science are slow and hard.
So is this really a crisis?
Maybe? That definition leaves lots of room for loose interpretation. But it really is no worse now than it has been in any recent history.
Would we all benefit from better understanding, higher discipline, and fewer supposed scientists taking short cuts? Of course, but when would that have ever been different? I am sure that predates the computer age by a good margin. By that definition — humans are in perpetual crisis.
Is it possible that the central planners of academia have given us a research and publication model that is broken at its very core? That we should stop creating incentive for publication completely detached from real outcomes? That excites the philosopher and the information engineer inside me — but is a far stretch from the purpose of this 5 minute article. If only Genevera’s presser had taken her there…
Or perhaps it is actually getting worse…
Not human laziness. Not lack of discipline. Average understanding — the recognition by everyday people that these systems are flawed… heck, broken.
With each increasing layer of technology, with each new branding & buzzword, are average people becoming more accepting of … let’s say non-facts? At a minimum, you can easily see where this may be causing more separation and uncertainty. Maybe we should blame market economies and sales teams for all this lack of transparency? But I digress… as I often do.
Or maybe it is just a matter of time…
Lately, a large number of people have become far more critical of the media. But so far, we really haven’t seen the same result for science. Most people still don’t realize what a debacle the “peanut allergy” crisis was. They don’t know why caffeine was public enemy number one — until it wasn’t. And they don’t remember “cold fusion” — because the system actually worked right there.
Genevera is on to something. But IMHO, the wrong something. The system needs an overhaul, but it has for a long time. For now — stay informed and keep a critical eye. Two things that are easier to do when you are calm. Just remember — Nothing smart was ever done in a crisis.
Oh, and thanks for reading!
