I think you are right to point out the impacts, but I think this also leads us to miss the real hazards of hate speech bans. I don’t believe in the principle of free speech as many do. I am not a free speech absolutist. But our extreme protections in the US come out of an important history, and while we should not be blind to present asymmetries, we should not forget the past or we will relive it badly (paraphrasing Burke, Santayana, and Henry Spencer). Additionally there are other similar problems which go the other way as well.
Indeed it is for the reasons you specify that I have come to agree with Justice Thomas’s dissent in Virginia v. Black. As you may recall, Thomas alone felt that the unique history of cross burning allowed the state to presume that it was intended as a threat, as long as such a presumption was subject to rebuttal. The other justices felt that the state could not presume such at all because cross burning fell under protected speech.
Rights and History
Our free speech protections in the US are the result of approximately a hundred years of judicial responses to abuses by the federal and state governments. What started when a man who advocated resisting the draft in WWI was prosecuted for sedition culminated in protecting even the abstract advocacy of genocide at a KKK rally. In essence in Brandenburg it wasn’t really that Brandenburg’s speech was in principle protected but that the government had shown themselves unable to draw reasonable lines and therefore the court had to draw lines with broad strokes.
This is further complicated by our experience with race in the US, particularly where the Japanese are concerned, and the fact that the 14th Amendment was intended to ban laws applying to one group but not another group (so-called “caste law” in 19th century parlance). Because there is no other way to prevent DC from being able to segregate schools, this same attitude is now read into the 5th Amendment as well.
So our rights are a product of history and of struggling with many of these issues. And so free speech laws in general are expected to be neutral and afford sufficient protection that the government cannot abuse rules to go after, in particular, the Communist Party. That’s the over-riding concern.
When I look at the Jyllands-Posten riots and the background of these a second concern comes up regarding both neutral and category-based laws. In this case, the Danish authorities concluded that speech that Muslims in Denmark felt was demonizing them was not worthy of prosecution and they prepared a dossier which was distributed around the Middle East causing riots. But if someone had similar pictures alleging Judaism was of the same nature, these would have been prosecuted and so Muslims in Denmark felt rightly singled out for less protection than other minorities.
Now, an argument could be made that Jews deserve additional protections in this regard due to the history of the 20th century, but the usual defense is that Judaism is deeply tied to ethnicity and therefore you have ethnic hatred issues raised as well.
The problem with that is that such a line in fact shows that no category-based rules really are neutral because unprotected categories overlap to differing degrees with protected categories. Certainly discriminating against non-felons in the US would not pose racial discrimination problems but discriminating unreasonably against convicted felons would. This is why category-based protections necessarily fail to offer equal protection and some groups are left out in the cold.
In one of Aesop’s fables, a wolf wants to eat a lamb and comes up with various excuses to do so. At the end, the wolf gives up on justifying himself and eats the lamb anyway.
We ought not to be under any illusions that all laws will be used by those in power to maintain or expand their position. This is true of both neutral and non-neutral laws alike. And as long as we want fairness from the judicial system, we will not get it.
The solution I think is to give up on assuming that policy, social machinery, and courts are going to solve these problems. The real problems are those of a lack of general solidarity. Therefore it is more important for people to come together and make common cause than to expect that they can find fairness in a judicial system run by impartial folks who only hear second-hand (at best!) selections of what is actually going on.
Common cause makes for common culture, not the other way around. And yes, that means that we have to afford space for groups we vehemently disagree with.