As a non-American, I don’t agree that free speech in and of itself should be absolutely protected. Yes we need freedom of expression to point out failures of society and give a voice to those without an audience. Further it is critical as you mentioned to discuss important issues and encourage open-minded discussion. However, hate speech or offensive speech as you put it in many cases does not move society forward in any way, but rather can inhibit its growth. The prime example being evangelistic religious groups that preach intolerance and oppression towards different-minded individuals.
So what could be safe about censorship? Protection of vulnerable minority groups should be an obvious enough reason.