The more that free speech proponents ignore the targets of white supremacist speech, the more complicit they become in their victimization.
Here you’ve committed the fallacy of the false dilemma, the same logic Bush used to get people on board with his dumb wars. “You’re either with us, or you’re with the terrorists.”
Nobody likes skinheads, or cults, or extremists, but they are allowed to exist in a free society. They are a byproduct of having a wide interpretation of freedom of speech, and I would have it no other way.
What’s your plan? Letting the flavor-of-the-day government move the goalposts of official censorship every 2 or 4 years, depending on who is in power? The government sucks at just about everything, so you should fully expect it to suck at deciding what is acceptable speech. That is just nobody’s determination to make in a free country.
Violence is a totally different story. If someone uses violence against you or your property, I fully encourage you to respond with force.