I think you’re basically right. The one exception I can imagine is something like an Interstate Highway System for self-driving cars, especially if they are able to run at speeds substantially faster than human-driven vehicles. Speed differences between vehicles on the same road are a big problem.
In the US, it might be hard to find land to build a second Interstate Highway System, but maybe in China or other countries that are less-developed.
In cities, I doubt human drivers will ever be banned, much the way bikes and pedestrians and even horses are not banned today.