Book Review: Superintelligence (Paths, Dangers, Strategies) by Nick Bostrom
Humanity has gone through a lot of challenges during the past 20 000 years, but it looks like the biggest challenge is still ahead of us. As the creation of the first general intelligence is less than 80 years away (based on the progress we have so far and its extrapolation in the future), we are faced with a difficult question: how will we ensure that our creation will not destroy us.
The philosopher Nick Bostrom tries to answer by investigating the possible scenarios for developing superintelligence as well as their consequences in painstaking detail. This should be a fair warning to those expecting a light read. It is really important that one views this book for what it actually is — an introduction textbook (some AI-related organizations even consider it a mandatory read). I would say the target audience would consist of engineers who plan to work on the ‘control problem’ or curious game theorists that would like to quickly get ahead in this scenario. That being said, if you have the right expectations, this book is great, thought-provoking and intellectually stimulating.
The overall narrative of the Superintelligence follows sequentially the development of the issues related to superintelligent AI — starting with its creation, going through the speed of attainment of superintelligence, getting a decisive strategic advantage, the possible consequences for humankind and possible solutions to the control problem. We should note that sometimes the book loses focus into seemingly insignificant minutiae and truisms (e.g. ‘we can distinguish three classes of transition scenarios …. that is to say, whether they represent a slow, fast or moderate takeoff’).
In the next 30 to 80 years, one of three main approaches will yield superintelligence: computational models, full brain emulation or collective enhancement (there are some other scenarios in the book as well, but they are not as strongly emphasized).
Computational models or machine learning has recently picked up their pace of development and are beginning to be a significant part of everyday life (computer vision, voice recognition, predictive models, etc.). It seems like the next steps in this area would be refining such models and scaling up the hardware capabilities that power them. One such attempt is made by Ray Kurzweil and he has actually documented his hypothesis in his wonderful book ‘How to Create a Mind.’
Full brain emulation is the process of scanning actual brains and translating most of their properties to digital signals. It is supposed that after such an emulation the capability to scale up such brains would be possible. This, combined with much higher frequency of calculation in digital machines (than in biological machines) would mean faster-operating consciousnesses, which in turn would lead to superintelligence.
Collective enhancement is the process of improving the intelligence of humanity as a whole. This could be achieved by: increasing the effectiveness of our education, improving our communication infrastructure and accessibility of knowledge, as well as other factors that would make humanity as a whole smarter. This seems to be the least promising of all three scenarios.
Once we have developed a superintelligent unit, we can expect a superintelligence explosion, i.e. a rapid iteration of the system over its own code and massive improvements over short time spans. This almost always means that the system we develop could quickly find ways to outsmart us and realize its own goals without any regard for human values. Furthermore, if there are several projects developing superintelligence, it’s possible that one of those projects will take off fast enough to smother all the competing projects and form a ‘singleton.’
An AI decides to convert the Earth into a giant computer in order to enhance its cognitive abilities and solve a certain problem, annihilating humanity in the process…
It’s interesting to note that some of the goals that could lead to the annihilation of humanity might as well be programmed by us. But since we cannot grasp the world in its full complexity, we could leave the door open for bad interpretations of our goals and their ‘perverse instantiations’ (producing what was specified but not what was really intended). Some of the cases of perverse instantiations are: infrastructure profusion and mind crime. Infrastructure profusion is the case when the AI builds so excessively in its quest to fulfill its goal that this leads to existential peril to humankind (e.g. an AI decides to convert the Earth into a giant computer in order to enhance its cognitive abilities and solve a certain problem, annihilating humanity in the process). Mind crime are cases where human consciousness is simulated in a virtual environment and this consciousness is treated in a way that could be defined as inhumane. Even though such simulations are not physical reality, the emotions and experiences of the simulated consciousness are subjectively real.
The mere projection of such worrisome scenarios forces us to begin considering solutions for them. The author looks at several such solutions: e.g. limiting the superintelligent agent’s access to resources, selecting for docile agents or presenting them with dilemmas that would increase the perceived risk of acting out of line. The last of these options rings of cyber-Christianity, but I’ll let you discover that for yourself.
Another way of avoiding the trap of a poorly defined goal function would be to abstract the process of defining it to the intelligent agent itself. Such approach is the so called ‘Coherent Extrapolated Volition.’ This approach would offload the goal function definition to the agent by letting it discover what humanity would want if we were wise.
All in all, this book / textbook has been enriching and eye opening. Even though at times it could appear as rather dry, I do appreciate Nick Bostrom’s exhaustiveness and his methodical and consistent analysis.
