I agree on generalities, but the devil is in the details :).

As you noted, main issue with continuous learning is how to define general fine-grained criterion for selective learning / forgetting. Such selectivity is a key for scalability: a current show-stopper for unsupervised learning.

I think universal selection criterion is a deeply theoretical problem, and theory is sorely missing in current ML, including DL.

This is my solution, I hope you find it interesting:

www.cognitivealgorithm.info