You said: “Just like a modern browser can read a web page from 20 years ago (had we gone with XHTML2 that would not be true).”
I tend to disagree. If browser vendors would have implemented an XHTML 2 parser, they wouldn’t have thrown away the old HTML parser. Browser would have come with two parsers on board: an HTML parser for text/html documents and an XHTML 2 parser for application/xhtml+xml documents. This way, modern browsers would have been able to read ancient web pages forever.
Not being backwards compatible was a rather smart move of XHTML 2, IMHO. This allowed for corrections of language design mistakes like alternative text being an attribute to the img element instead of element content. Unfortunately, this correction cannot be made in HTML5.