Yes, there are some restrictions on our free speech rights, but they are few. And we accept them as reasonable.
First, there’s slander and libel. You probably won’t get charged with a crime if you willfully libel someone, but the person you libeled can certainly sue you for damages, and claiming “free speech” to deliberately destroy someone’s reputation with lies won’t fly.
Second, there’s speech that puts someone in imminent physical danger. This is the whole “shouting ‘fire!’ in a crowded theater” thing. Speech that is intended to start a stampede or incite a riot isn’t a protected right — although the line is blurry, and each case is carefully considered as to whether the First Amendment applies or not.
These are common sense practices designed to protect people from harm. But note that these are for extreme situations, and have a fair number of requirements. For example, for a libel claim, the libeled party must have incurred some kind of injury or harm. It doesn’t apply if your sister calls you a poopypants, for example.
Once we start expanding these exceptions to include bans on calling for boycotts, can bans against criticism of our government be far behind? Let’s not start on this slippery slope.