Area4: A peculiar danger of automation

Jan 18 · 2 min read
Does automating motorcycles make sense?

You might think that automation is always good. You might even think that the more automation the better.

I don’t blame you. I used to think that too.

As I have briefly discussed before in many situations automation might be dangerous. It’s the peculiar spot of being 90% there.

In my experience it’s better to have 50% of the job automated than have 90% automated.

As mentioned in the article above, Tesla pushed automation HARD in their GigaFactory. Some of the workflows were really difficult for the machines. The automation there was 90% done. The problem is the remaining 10%. In the GigaFactory example, that caused massive delays. Mistakes made by the automation robots stopped whole assembly lines. There were not enough people to absorb the fallout.

A similar problem on another level is observed in single human interaction with automated systems.

Human factor experts have known for years that very highly skilled personnel do not do a good job of monitoring for events that have a very low probability of occurrence, those who maximize the use of technologically possible automation often miss this very critical point.

This happens for example in the cockpits of aeroplanes. The plane can start, fly and land by itself. Literally, it could be that the pilot only sips the single-serving whiskey the whole flight. But no, the computer systems in modern planes are programmed to ping and probe for reaction from human operators so their attention is sharp.

This is not such a big problem with the military-like personnel like pilots, even of the civil kind.

Recently the biggest problem of this sort is the car autopilots. They are, again 90% done. Manufacturers repeatedly inform users “You need to pay attention, this is only an aid”. But people, obviously, ignore it. Because it’s so good. It gives you the illusion you don’t have to do anything. Over time, with nothing happening you progressively lose your concentration. That’s a dangerous place to be in.

On a larger timescale, when car autopilots become the norm. The whole skill of driving will meet with the same faith as people’s concentration now. It will slowly fade away. Driving will become an eccentric skill to have.

This article also appears in:


Written by


Automation believer and enthusiast -