Perhaps there’s another reason that it’s always men, that doesn’t have to do with power so much: Men are culturally trained to be the pursuers in romantic relationships (and honestly everything else), while women are generally trained to be pursued. Maybe the reason that men are always the ones trying to initiate conversation is because we’re expected to, whereas women aren’t.
I’m not trying to reduce the impact of your thoughts here, just offering an alternative hypothesis. While I am a guy and don’t generally have this problem, I can imagine that it is annoying to have to wave people off day after day… even a short polite response which would take no time and be little effort once can scale to a sizable annoying problem. Personally, I think earbuds in is a strong signal that you want to be left alone, but that could be because I do the same thing.