“then it’s not exactly ‘oppression’ but it’s extremely unpleasant.”
It can be oppression if the intent is to slander someone as a racists/transphobic/homophobic/sexists to stop people from listening to them, or at least vocalizing their agreement. What’s more is you can further oppress someone by shaming them for thinking they haven’t done anything wrong.
Oppression is simply making sure you maintain control over other(s)
If you can successfully label someone as racists/transphobic/homophobic/sexists you can cut them off and prevent them from being listened to or reasoning with others before what they’re saying can be accepted. It has the added bonus of making an example of someone so others will do as you say.
It’s such an effective means of control, people are even afraid of just the accusation. Most won’t even talk to people that get labeled as potential <whatever> to even see if the accusation is warranted. You’re reputation (personal and professional) is tarnished by even trying to find the truth and not just taking the accusers word for it. That seriously impacts jobs, families and all kinds of other relationships, just for not outright condemnation. Thus, being the accuser gives you control over people that otherwise could/would have large influence.
That’s not to say some people actually aren’t racists/transphobic/homophobic/sexists, but when the labels get applied to people because of a shirt to force them to apologize (for a shirt), or someone making a dumb joke with a friend (to have them fired), or taken out of context to frame a very well respected mentor (forcing them to resign). It’s hard to see those labels as anything BUT oppressive.