Do you think we can do this with humans?
>I don’t think that an AI should unpack the entire action a or state s
Wei Dai


To be clear, we are comparing “Paul evaluates an argument for X hours” with a recursive procedure where every computational step is determined by Paul thinking for X hours. In general it seems intuitive that the latter is going to be way better, which you aren’t necessarily disagreeing with here.

Your concern is that the only way to make sure it is really better is for one of my recursive invocations to look at the whole argument, that is that there are some arguments for which there is no way to do any useful abstraction or subdivision of the argument.

If I think about philosophical arguments that I have ever seen, it feels easy to do some subdivision (where the subdivider does not themselves have to look at the full argument). For example, many arguments involve several claims which can be preprocessed and analyzed, separate steps of inference that can be analyzed separately, introduce individual concepts or definitions that can be processed separately, and so on.

Even if such subdivision were to introduce costs, they feel very small compared to the direct benefits of the recursive procedure, in the sense of the recursive procedure more than fixing all of the particular flaws introduced by the subdivision.

So in the context of existing arguments it feels very likely that we could do this. When I think about the space of all possible arguments, and all possible ways in which our judgment could deteriorate, I agree it is less clear.

Also: if we are talking about arguments such that merely looking at them drives the reader insane, then even mild preprocessing of the definitions or concepts seems very helpful (this is related to my response to your first paragraph). And if we are talking about arguments such that processing them in detail causes trouble, then you can have steps where you look at the full argument so long as you don’t try to process the whole argument as a single unit.

Like what you read? Give Paul Christiano a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.