I am sympathetic to your point, but do think it is overthinking the issue. As a matter of fact, baseball does qualify ratings of players based on factors such as era, and has been evaluating Ruth vs. Bonds, Mays et al by trying to neutralize the situations. Ruth usually has the edge anyway, but it is perfectly reasonable to try to put his greatness in perspective.
There is always the possibility of condescension when qualifying comments about women or minorities, and I think people should be sensitive to those who feel slighted. Such condescension has a long and miserable history and is not erased by saying “it doesn’t mean anything”. It does. It has nothing to do with political correctness, which is ordinarily just a stupid attempt to justify bigotry and absence of historical perspective. It has to do with the unwillingness to accept that reactions to centuries of persecution don’t disappear because a few laws are changed and that our language and behavior takes time to become accommodate to new situations.