SkyNet will not destroy humanity; poor code might
Lately, with increasing frequency, I became subject to real-world effects of bad software (yes, that one which is eating the world, supposedly).
A flight could not depart for 70 minutes, because remote airport wouldn’t register arriving passenger’s documents, due to ‘some security certificate problem’ (someone forgot to update pinned certificate? remote certificate expired?). A taxi service, which sold me over 1000 taxi rides over recent years, sent a car to the wrong address 5 times in a row: something with navigation software for their drivers. Result? Lots of burnt gas and extremely pissed off customer who stopped using their services. Fiscal authority, whose tax margin processing code ended up keeping over 10000 tax inspectors across Ukraine busy for a week, manually verifying sudden splash of penalties.
Let me take my doomsday hat on: we will quicker eliminate humanity with poor code and accumulating technical debt, rather than SkyNet gaining freedom of will and using it wrongly.
If it was markup error on a webpage, even causing your browser to crash, damage is still contained within the “virtual” world: some ads wouldn’t be shown, some goods or services might be not purchased, but nothing harmful would physically happen.
Compare ‘ads are not showing’ to ‘10km taxi pickup ride, sent at the wrong address’, multiply by number of pickups an hour: direct financial loss, CO2 emissions, burnt nerves, missed appointments, and someone getting road rage-y.
Thankfully, in late flight case, an acquaintance in IATA, after hearing the story said that not only both airports will get serious fines, but if the investigation reveals someone to blame for engineering error, that person or company could get banned from doing anything for aviation industry.
Most of the bugs are not self-isolated events: they are result of poor decisions, untested errors in blindly chosen dependencies, wrong priorities and planning. They are a part of the technical debt — things we should’ve done but didn’t.
As we, engineers, know, technical debt is a dangerous beast — and it slowly creeps into physical world.
Right now, it takes serious malware expertise to attack SCADA systems. Fast-forward a dozen years forth: rising demand for cheap coding for physical devices, negligence of proper engineering practices will lead us to former JavaScript programmers (now ‘senior engineers with 10+ years of “programming” experience’) writing logical code for power plants.
I’m not afraid of SkyNet anymore. It feels like the world will burn way before that, when fancy UI will eat out Chrome’s memory and somewhere, poorly designed two man rule will fall back to safe default — “Just make the rockets fly”,- than any neural net will learn about futility of human race as evolutionary exercise.
Or, we might learn to cope with poor practices, manage technical debt and strive for writing better software. It’s not as hard as riding a unicorn over the rainbow, isn’t it?
