Have you ever had the minority opinion? Have you ever been in the wrong?
Absolute negative rights exist to protect people from the majority, from the authority, regardless of who that majority is now and in perpetuity. If you don’t see the value in that you are the majority, you are the authority you are the one punching down. And as history shows, times and majorities change, attitudes regarding principles not so much. When society was profoundly transphobic someone with your level of intolerance would have made your life hell or probably ended it. Careful what seeds your behavior and attitudes spread.
A society for all is a society where you have to tolerate absolutely. You let bad ideas become jokes and fade into irrelevance. ANd yes, I said jokes, you don’t ban them because out of conflict you create camps and camps create advocacy. If you feel Spencer or white nationalism is relevant you will make them relevant. If you draw a line in the sand and insist that anything other than progressive speech is extreme speech, you just created a wind mill monster and created a societal dichotomy where you are the agressor and white nationalism is the victim people will defend.