we don’t know how we do what we do because it’s happening automatically and unconsciously.
That’s the crux, and yet the author reaches the opposite of the conclusion that follows logically. More specifically, this is about emergent behavior in complex systems — a phenomenon that, among other things, turned a formless, dumb mass of particles into the universe we call home, including our conscious brains and minds. While it took billions of years, it happened indeed “automatically and unconsciously” — just as a lethal AI could emerge spontaneously out of nowhere. Most likely we won’t know it’s coming and we won’t be able to control it, whatever form it takes. Of course, it could just as easily be benevolent or benign, or perhaps even something that our limited intelligence cannot perceive as intelligent. Maybe it will wipe us out without being aware of us at all, as in some sci-fi stories where vastly superior aliens didn’t even notice human ships attacking them. Or maybe nothing at all will happen.