I think the existing systems you are talking about are based on imitation not because imitation is an inherently powerful approach but because they are solving problems that lack precise specification and imitation is the only feasible line of attack. Indeed, as an expert in computer vision my experience is that all problems that have more or less precise specification (such as navigation or 3D reconstruction) are much more successfully attacked by “non-AI” algorithms. This is also the reason most progress in game playing AI was by approaches other than pure imitation (i.e. approaches that involve an explicit analysis of the consequences). Moreover, much of the success of deep learning is thanks to moving from pure imitation to taking the structure of the problem into account, that is, looking for features on different scales (but I’m not an expert on machine learning so take the last observation with a grain of salt).

I also think there is substantial difference between imitation and approval-directed agents. The former produces outputs that a human would produce. The latter produces output that a human would *approve*. The latter is a much harder problem as we know from the P vs. NP conjectures. Conversely, the ability to solve the harder problem is much more powerful.

The only difference of principle between goal-directed agents and approval-directed agents is that they maximize functions of different type: the goal-directed agents are maximizing something that depends on the consequences of executing a plan and the approval-directed agents are maximizing something that depends on the consequences of the inspection of the plan by the overseer. However to reach self-improvement we need to maximize the ability of the successor agent to maximize, taking goal stability along future generations into account. This seems much closer to goal-directedness than approval-directedness.