Resource: Superintelligence (Nick Bostrom)
Foundational reading for understanding potential paths to and from AGI’s eventual takeoff
I’ve been plugging away at Professor Nick Bostrom’s Superintelligence for a few months now, and must say that it feels like an accomplishment finishing it. Don’t get me wrong, it’s a worthwhile, thought-provoking read, but wading around in hypotheticals takes some getting used to.
The best way I can describe this book is like an ever-expanding decision tree, that adds new branches as you read. The progression of chapters is intuitive and the structure in each chapter is clear, but within each section, you’ll find yourself exploring myriad coulds and mays. The web of connections that form between these new and inconclusive thoughts— particularly late in the book when layers of concepts are combined — shows the immense complexity of managing the rise and effect of superintelligence.
Professor Bostrom is the first person to acknowledge that he, nor anyone else, is yet prepared to deal with all the uncertainty. He begins with:
“Many of the points in this book are probably wrong…I have gone to some length to indicate nuances and degrees of uncertainty throughout the text — encumbering it with an unsightly smudge of “possibly”, “might”, “may”, “could well”, “it seems”, “probably”, “very likely”, “almost certainly.” Each qualifier has been placed where it is carefully and deliberately.”
By the time you complete the second chapter, you’ll appreciate the need for cautious wording. Through the subsequent 13 chapters you’ll gain an appreciation for how much we cannot yet predict and the degree to which we’ve theorized about it. Most of the time, you’ll be grasping for something solid to hold onto.
To Bostrom’s credit, he repeatedly attempts to frame potential outcomes on their level of desirability when he can — even if it’s only directional at this point. The real value of walking through this tree of outcomes with him is not to understand the ‘best’ path or the most likely one, but rather to discover and learn about each path.
Each chapter explores a critical concept and they all feel important (at least to a beginner like me). That said, here’s a sample of a few I found particularly enlightening:
CHAPTER 4 — The kinetics of an intelligence explosion
An effective summary of the forces at play that create rapid, exponential acceleration of intelligence that’s hard for humans to comprehend.
CHAPTER 8 — Is the default outcome doom?
Particularly the section on malignant failure modes, that explore several unintended outcomes of actions with harmless intentions.
CHAPTER 9 — The control problem
Understanding control methods and motivation selection issues felt like mandatory reading here. Motivation selection particularly sets the stage for issues in the second half of the book.
CHAPTER 13 — Choosing the criteria for choosing
It’s fascinating thinking about what we’d identify as the machine’s intent even if we could control it. Ditto for how a superintelligent machine may be better than we could be at defining that intent safely.
CHAPTER 14 — The strategic picture
Bostrom gets to how we should consider going forward. His exploration of how competitive/race dynamics may impact safety tradeoffs really underscored how damaging a lack of collaboration could be.
Having read the book, I feel more informed, but not more certain about what will happen. In many ways, this is the book’s purpose: uncover issues and possible outcomes you hadn’t considered and let you know the stakes are unusually high. This is well encapsulated here:
“Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct. Superintelligence is a challenge for which we are not ready now and will not be ready for some time.”
His metaphor continues, but his message about our inability to grasp the severity of the downside here is clear.
Bostrom acknowledges in his 2015 afterword that some (read: many) viewed this book’s focus on risks as his belief that things would go wrong. He refutes this, but does a make this salient point:
…At this point in history, whereas we might get by with a vague sense that there are (astronomically) great things to hope for if the machine intelligence transition goes well, it seems more urgent that we develop a precise detailed understanding of what specific things could go wrong — so that we can make sure to avoid them.
I’m increasingly on board with this sentiment. I’m not suggesting progress in this field’s advances should/can stop (nor is Bostrom as you’ll read), but that we should accelerate investment and collaborative efforts that mitigate risk in proportion to the severity and likelihood of the downside.
That’s my conclusion from this read, and why I’ll continue recommending this as a resource to calibrate how cautious and certain we should be about the future of AI.
