They’ve succeeded at associating with you, because the only thing that makes you “you”, is something anyone can do in a manner of seconds with no approval process.

If a company’s spokesperson, a member of a political party, or really any member of any organization, goes off the rails and starts saying or doing horrible things, that company, political party, or organization can kick them out. They can fire the person, revoke their membership, or (and sometimes this is the case) do nothing if they think they’re better served by sticking with the person. It doesn’t really matter how they respond, the point is that they can.

You’ve got every right to be frustrated that a choice you made for entirely sensible reasons has been co-opted by trolls who are now making you look bad. But because that choice has no gatekeepers, you have no recourse other than to stop doing the thing you have in common with the trolls. That’s not fair, but it’s reality, and complaining that it isn’t fair will do nothing to change that reality.