Nice post — well written. The silver lining is that as the time passes Selenium is becoming part of the web standards — and the drivers are developed by the vendors. The problems you describe will remain of course for old browsers. But it’s a good bet to stick with Selenium.
May I suggest that, given all the efforts you already spent, you try to add visual validation to your test suite? I like to say that an automated e2e test takes you to the interesting places in your system — and, as in real life, if you got to an interesting place — why not take a picture?
The advantage of adding visual validation (i.e. comparing the screenshots taken during the test to the ones from a previous run) is significant: in a regular e2e test you search for bugs that you know of (e.g. — you click ‘+’ and assert the counter’s value because you know it tends to be buggy). In a visual test you stumble upon bugs you weren’t expecting because you take a screenshot of the entire page/viewport and you’ll get a diff alert if any visual regression occurred.