I think the diversity movement will be stronger if it demands inclusiveness in hiring and corporate cultures welcome to all. Pitting one group against another, or ranking grievances, is simply not constructive. If companies wholeheartedly adopt the mindset of diversity, everyone will benefit.
In any case, you note, age is a bias that also includes people in groups already underrepresented. I’m totally down with comments you make in your last paragraph. No matter what your gender or race, you get older. (Unless you’re really unlucky.) As you say, everybody deserves a chance to be judged by “what they can bring to the table.”