You are right that A1 ought to be able to win using only a powerful computer, since that computer…
Paul Christiano

Another argument along the same lines is that there is clearly some minimum capacity that the base agent must have in order for recursive bootstrapping to work (i.e., reach arbitrarily high capacity). I think the smartest and most philosophically inclined and sophisticated humans might barely have enough capacity for this, but here you’re proposing to bootstrap not the human but an AI with less capacity than the human, so it seems unlikely that the AI would meet the minimum threshold.

Like what you read? Give Wei Dai a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.