I think that is the crux of the issue. Definitions. How do you define hate speech?
As far as ‘harassment’ — I think that has a long history and is easier to define. Threats to your person, stalking, menacing, showing up at your work uninvited, calling up your family and alluding to threats. That sort of stuff.
And harassment, on a whim, without research, I’d roughly define as: A pattern of behavior and/or speech meant to intimidate, threaten, menace, or prevent your free exercise of movement and speech. Not limited to physical violence or acts (though cross burning in your front yard seems like an excellent example of harassment) but also long-term directed and personal speech (e.g. “I know where your family lives,” “How much do you love them?”, “Do you have a will written?”, “Have you ever thought about how hard it’d be to lose a family member?” — statements that wouldn’t constitute assault but most reasonable members of society would find threatening when taken together).