When Software Kills
Software bugs, errors, and oversights have been blamed for hundreds of deaths in recent history. What are the takeaways for software engineers?
The preliminary investigation into two fatal plane crashes involving Boeing 737 aircrafts has brought to light that the company’s proprietary software may have been at least partially responsible for the crashes, in which 346 people were killed.
This is not the only time in recent history that software bugs, errors, or oversights have been blamed for catastrophes. In some cases, hundreds of millions of dollars were lost. In other cases, lives were lost.
How do we ensure that safety-critical systems are supported by reliable software? Is human error always to blame in the event of a software failure? What are the takeaways for software engineers?
A brief history of catastrophic software bugs
Between 1985 and 1987, at least 5 patients were killed (and others critically injured) when a software-controlled radiation therapy system, the Therac-25, inadvertently administered massive overdoses of radiation — over 100 times the prescribed dose.
A similar event happened in 2000, when software written by the American company Multidata caused dozens of Panamanian patients to receive huge overdoses of radiation, 5 of whom died as a result. A total of 9 affected patients died over the next few years, likely as a result of the same overexposure to radiation. The bug happened when different doses of radiation were administered depending on the order in which data was entered into the system — an error that was exacerbated when doctors looked for loopholes to adapt the software’s usability. The doctors who failed to double-check the software’s calculations were indicted for murder.
In 1991, a ballistic missile struck a U.S. army barracks in Saudi Arabia, killing 28 and injuring 96. A software error prevented the missile from being intercepted.
In 1997, a poorly programmed ground-based altitude warning system was deemed partially responsible for a Korean Air crash that killed 228 people. The bugs in the system were acknowledged and corrected by the FAA. Although…