“In order for these systems to behave wisely, the user has to actually be wise. Roughly, the users need to be intellectual peers of the AI systems they are using.”
I don’t get the intuition behind this. Humans are more able to change the environment than to predict exactly what change they will make. This makes me think that it is not enough to be the intellectual peer of the AI system to avoid bad outcomes.