You are right that A1 ought to be able to win using only a powerful computer, since that computer…
Paul Christiano

How does A1 engage in a very extensive process of reflection, given that it presumably doesn’t trust running itself at capacity higher than c1? (In the human upload analogy, perhaps A1 fears going subtly insane if it runs for more than a subjective day without being reset.)The best that it could do seems to be to spawn multiple instances of itself, each writing down its thoughts for the next generation to read, but how do we know this produces the same outcome as B0 himself going through a very extensive process of reflection? It seems likely that the distributions of outcomes these two processes produce would be rather different.

Like what you read? Give Wei Dai a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.