Another argument along the same lines is that there is clearly some minimum capacity that the base…
Wei Dai

Note that the AI can consult the human during the bootstrapping process. I think/hope that if the human clears this minimal threshold, then the process can be fine. (The AI still needs to clear some minimal threshold, but it is more closely connected to understanding how to consult the human rather than encoding all of the human’s knowledge about how to reflect.)

Note that the overseeing process need not be a human, it can be any physical process we like (a group of humans, human with tools, whatever).

Show your support

Clapping shows how much you appreciated Paul Christiano’s story.