Exactly. Nailed it.
Speech is the vocal expression of thought. Should the expression of an idea trigger involuntary action, all communication would need to be carefully moderated. But taking offense is voluntary just as taking action is voluntary. You have to decide to act, and the consequences of your actions are always on you as a willing and informed participant.
Ideas are invisible, imaginary, and ephemeral. They don’t exist outside the mind. You cant defeat the intangible with the physical; ideas can only be defeated by better ones replacing them in our collective headspace.
Taking action is tangible, and affects the physical. That’s why free speech is permissible and why free action is not. And you don’t get to blame your voluntary action on the ideas or words of others and absolve yourself in the process. We haven’t outlawed thought (yet), only socially unacceptable actions.