Roko’s Basilisk and AI Decision Making

Caution: reading this may commit you to an eternity of suffering

Ed Noble
Thoughts And Ideas

--

Photo by Nivedh P on Unsplash

In May an official from the US military gave a presentation on the future of autonomous weapon systems. Col. Tucker Hamilton, the Chief of AI Test and Operations, is involved in the testing of flight systems, including robot F-16s that are able to dogfight. In his presentation Hamilton spoke about a simulation in which a drone controlled by AI was tasked with identifying and destroying ‘surface-to-air missile’ (SAM) sites, with a human controller giving the final ‘go’ or ‘no go’ instruction. At times, the AI would identify a SAM site but the human would give a ‘no go’ instruction; after a while the drone decided to kill the human operator, as it was interfering with its higher mission.

Hamilton went on: “We trained the system — ‘Hey don’t kill the operator — that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.” I should note that the US military later confirmed that this was a thought experiment which was anecdotal, rather than an actual exercise, and no one was hurt.

However, what it shows is that for an AI, no action is considered off limits because of…

--

--

Ed Noble
Thoughts And Ideas

I write about philosophy, psychology and ethics. I live and work in London, having previously studied physics. Started writing in lockdown.