They are driven by all manner of drivers with all manner of qualifications with wildly varying degrees of attentiveness and ranging in age from 15 to 95.
Self-Driving Cars: Have We Completely Lost Our Minds?
Terence C. Gannon
6928

And this is precisely the point. Do you realize that your argument against self-driving cars mostly relies on the incompetence of human drivers?!!? What if we thought of self-driving as like vaccines: primarily effective after a saturation point? Even on your own argument, a world with mostly self-driving cars is vastly safer (and just more pleasant — shorter, more fun commutes etc.) than a world with mostly drivers; the key trick is just reaching that saturation point as quickly as possible.