This is one of the best things I’ve ever read. As I’m writing a few argumentative pieces on A.I. and whether or not it should be regulated, and if so how heavily, I’ve seen all too many writers who have published a sci-fi novel thinking they are the final voice of all things AI. You can pick these boogey-man writers out because they always want to defend themselves, and have to explain the delusion that these books are realistic in anyway, and give them knowledge that no other human on earth has. These are the elite of the elite.
I wonder, however, if the world will ever decide on a constant definition of artificial intelligence. The article you included in this piece defines super-intelligence as ‘an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills.’ In no way is this realistic as a threat. This definition of super A.I. will surely ruin us as a species. If it becomes better at programming, then it will eventually know how to program something that won’t take it over. Let’s face it, we’re trash. A computer programmer doesn’t need to have the best social skills or the best general wisdom if it can create something that does. That’s why we as humans have computers anyways, to do a VERY specific job so we don’t have to.
When you refer to China not giving a ‘hoot’ about legislation, (although that is a form of boogey-man scare tactic you referenced earlier, moving on) I believe that every country will bend the rules as much as possible, even to the point of breaking them if the gain is greater than the punishment. The final paragraph in the cited article by Oren Etzioni mentions using AI to enhance biological human intelligence. If a country wanted to make its top scientists smarter, we would have to experiment with this, even though there is an extremely small chance that any UN or other global power would allow any testing of the sort. I personally would say screw it, let’s do this, and ask for permission later.