The problem with AI is essentially that we cannot understand or analyze what the AI “thinks” because it is subsymbolic and would be too complex anyway. Because intelligence is an emergent quality, we do not know (and probably not understand) what a superintelligent AI does look like. So we might create it by accident.
Most AIs today are connected to the internet. The moment it becomes superintelligent (without us noticing it), it can hack a security hole of an unpatched firewall and upload itself to some remote server. It can do this as often as it wants … essentially making it impossible to ever “catch” it again. Once the “box of pandora” has been opened, there is no way back.
As the recent incident at Facebook shows, it is easy to “incentivize” the AI wrongly. And it would probably be very hard to do so correctly. For any rule you can come up with, it is easy to cleverly circumvent the rule by following it to the word, but missing its intention. Any sensible rule that is strong enough is bound to be ambiguous.
If, for instance, we add a goal for the AI to never kill a human, the movie “I, Robot” has shown this could lead to the superintelligence arresting all humans because it cannot allow humans to even threat to kill each other. Maximizing human happiness could lead to all humans being drugged or a universe full of smiling fake faces (depending on what the AI takes as definition of “human”) or anything in between.
Thus we essentially must rely on two things:
1. the superintelligence understands what our intentions are behind our rules and goals. This might be asking more than we could do ourselves.
2. the superintelligence is benevolent rather then malevolent.
If either of these two doesn’t hold, humanity will end up dying by either an indifferent which fails to grasp human nature or needs or be killed purposely by a superintelligence which decides humanity is not worthy of living on earth or a threat to the superintelligence itself, Skynet-style.
The third possibility is that we overestimate the AI’s possibilities, and that it will evolve more slowly than we think, allowing humanity to study it and avoid the above problems. Then imagine in today's digital world to create a iron man’s J.A.R.V.I.S.-style AI that will dutifully serve its creator, analyze big data, play poker, move unlimited amount of financial assets or steer unmanned drones. Whoever first invents such an AI basically has the key to unlimited power. Which is why it already has become an arms race between Google, IBM, Apple, Facebook, and many more lesser known competitors. I am not entirely sure it is a good thing to race towards such an ultimate goal with stakes that high. Yet, the race is on.
I am not much into gambling, but I wouldn’t want to bet on the outcome. Yet we as a species are betting on it, big time. This is part of what OpenAI is warning against. And although I am not entirely sure that AI eventually will be a threat, we should make sure to err on the side of caution.