Equality, like freedom, demands duty and responsibility. It is something that should be assumed as a given, but when it is not, then people must be called to account. If one cannot treat others equally and fairly, then they must be set aside and empowered to expand their thinking and change their behaviour.
Freedom of speech is not an excuse for spouting out hurtful or hateful things. It is a responsibility to use our freedom wisely; if it is used for the violence of putting others down, hurtfully pointing them out or in any way denying their equality and full personhood, if not used responsibly, it should be taken away. Period.