Take Out the Machine Learning Garbage Before it Smells

Jack Raifer Baruch
The Startup
Published in
7 min readFeb 26, 2021

--

Machine Learning is an amazing technology. Unfortunately, its hype, like with everything that becomes “cool” or “in”, is starting to create a snake oil market full of smoke and mirrors, with potentially catastrophic consequences.

Apologies for the apocalyptic start to the article, but it is not wrong. Let me give you a little bit of background so we can dive into the details of the problem I am talking about.

Many, many (probably one more many) years ago, before I even knew what Machine Learning was (or even before more than a handful of people knew it existed), I studied psychology, and spent a lot of time working in the area, from psychometrics to organizational behavior and culture. In the psych industry, being science oriented can be seen as, and in many cases is, a hindrance. The reason being that the lack of scientific rigor in many parts of the industry is conductive to the spontaneous appearance of panaceas, mythical, magical, cure all solutions to all the pains of humankind.

For example, one of the cringiest pseudoscience’s in existence is the infamous NLP or Neuro Linguistic Programming (not to be confused with the actual scientific tools for Natural Language Processing). This supposedly amazing technique, can purportedly reprogram the brain and help you become superhuman, allowing you to tap into 90% extra potential of our brains (even though the brain does not have unused portions to tap). The problem is that the science behind it is, at best, badly crafted and terribly suspect, and at worst, pure and utter garbage. And yet, the NLP industry moves billions of dollars a year and has practitioners and masters all over the world.

Why would something that cannot be proven to work be so successful? Simple, it sounds scientific, it makes people feel like they can get better, and most of all, it offers to will work fast (if you accept that if it does not work, it is because you did not believe enough in it). This mixed with some motivational strategies right out of any cult playbook, and you have the perfect mix of smoke and mirrors to cash in. Even worse, since the advent of serious neuroscience, the NLP movement has exploded, hiding the empty shell behind scientific research, watered down, and realigned to fix their needs.

Eventually, like it has happened to all pseudoscience’s, it will die out, as new fads come into the frail and actual scientists move our knowledge forward. However, it does create problems, mostly because a lot of people that could have used actual psychological therapy, you know, the kind that has been proven to work, ended up wasting their time and money in snake oil. Even worse, after they discover it does them no good, instead of blaming the culprits, they end up blaming the industry, in this case, psychology in general. And at some level, I do believe us psychologists, as a community, do share part of the blame for not pushing hard enough against these kinds of (apologies in advance for the expletive) harmful bullshit.

Now, back to our regular Machine Learning programming. Once I got into data science, I felt a bit of relief from entering a community that is data driven and scientific oriented. However, occasionally, I do find that the same pattern repeats itself, and the smoke and mirrors come out of the woodworks, this time hidden behind a stage filled with algorithms and models.

The latest culprit is a startup in Munich, who apparently, have created an amazing machine learning model capable of objectively measuring your personality from a short interview video. This sounds amazing. If it really works.

So how does this incredible system do it?

It purportedly uses the scientifically robust OCEAN personality model (this psychometric model has A LOT of good research and evidence behind it), so it starts out well. Then, the subject records a video of them answering some basic questions, the same you would have on an HR department interview. Please do note, that at no time do they use any of the painstakingly crafted and tested questions from the actual OCEAN personality model questionnaire. And then, after running the video interview through the model: ta da, you get you OCEAN personality results.

Image from https://web.br.de/interaktiv/ki-bewerbung/en/

This sounds amazing. But when we start looking under the hood, we begin to see a few problems. The model was built with 12,000 videos, and the 2,500 rated their personality using the OCEAN model. According to the company, the model has a 90% accuracy when compared to the human evaluators. But here is a basic problem. The OCEAN personality model was never built for people to evaluate others by eying a few minutes of video, or even a short in person interview. It is an extremely specific questionnaire that is to be filled by either the subject, or people close to the subject who know them personally, never by an external observer, and much less after a brief interview.

But I could be wrong, maybe they hit on to something amazing. Until you discover that a group of journalists in Germany took it upon themselves to investigate the model in action, they even hired actors to play the part, and this happened. With small changes such as glasses or a headscarf, the results from practically the same video, with the same answers, change drastically.

Image from: https://web.br.de/interaktiv/ki-bewerbung/en/

Worse still, changing the background give drastically different personality results.

Image from: https://web.br.de/interaktiv/ki-bewerbung/en/

Even a small change in the lighting conditions, changes the results.

Image from: https://web.br.de/interaktiv/ki-bewerbung/en/

The company makes the excuse that these kinds of things, like backgrounds, glasses, hats, and the like, also alter our perceptions of people. This is true, sort of. First impressions do change a lot by a myriad of thousands of tiny factors. But we also know we are very incompetent at reading people, especially on first impressions. The reason personality models like OCEAN exist, is precisely because we need a measurement device to compensate for our inability to measure such things accurately ourselves.

Even if we allow for such careless explanations with a clear lack of domain knowledge: What use is a system that cannot give consistent results? Would you trust a scale that changed the weight of a product because the day was dark, or the buyer was wearing glasses? Of course not. Inconsistent measurement devices end up where this one belongs, in the garbage.

Even the journalists, although they did a fantastic job of bringing the issues with this model to light, made the mistake of framing the problem as one of bias. And yet, this is much worse than bias, this is outright trickery, flat out lies and an attempt to sell fools gold. The creators of the model clearly misunderstood, or conveniently misrepresented the science behind the OCEAN personality model and used its good scientific standing to create a worthless piece of technology, one that is hurting people since it is, today, being used to decide the fate of job seekers, using bad science and a worse application of machine learning tools to make money at the expense of peoples livelihood.

Is this the industry we want? One that allows for bad science, bad models and outright lies?

Can we allow a young discipline like data science, a new technology like machine learning, to be used by shoddy salespeople more interested in a quick buck then in the integrity of the industry?

We should not… We cannot…

We are scientists, we are data driven, and we must, always, call bullshit where and when we see it. Or else, we will rightfully be blamed for allowing charlatans to lead our industry.

The first thing you learn in Data Science is the now famous phrase: “Garbage in, garbage out.” This is garbage, and like the rest of the trash, it is our duty throw it away before it smells.

Hope we cross paths through our Journeys…

Jack Raifer Baruch

Follow me on Twitter: @JackRaifer

Follow me on LinkedIN: jackraifer

About this Article

It was inspired by the amazing work of a group of journalists in Germany. All images come from their article and the rights belong to them. Feel free to read their work on: https://web.br.de/interaktiv/ki-bewerbung/en/

--

--

Jack Raifer Baruch
The Startup

Making Data Science and Machine Learning more accessible to people and companies. ML and AI for good. Data Ethics. DATAcentric Organizations.