Great post! A perspective from someone who is not primarily a programmer, if I may.
Usually, step #4 (understanding why/how/where O !=O’ ) is where everything interesting is. Most people engaged in research do programming only as a tool in course of addressing other problems and we rarely go beyond adequate (perhaps better in the domains that we do frequent, but that’s only a small subset of all programming realms.) The useful feature of PbE, from my perspective, is that computers learn differently from humans, which, in turn permits useful insights by tracing how they learned what they learned — that is, what patterns the algorithm saw that were not obvious to the human. But to take this path, the important thing that we need to understand, I think, is not so much what the machine predicts, but how the machine predicts.
This is where a lot of buzz around DL bothers me. It is one thing that DL can predict a lot of things, but what mistakes/alternate choices does it make that a human (or another algorithm) would not, and what do those mistakes say about how the machine “thinks”? What perspectives does a machine add to a human thinking (or an alternate algorithm thinking), not so much that DL can do all sorts of things. These are not the questions that seem get asked often compared to the buzzwords.