For me, I made the rule when I realized what is at stake. My love for my child, my mother, and my sisters made it my duty. I think history shows us many examples of people or groups divorcing themselves from hurtful language as a means of self-preservation.
To better answer your question though, I belive that whoever is harmed by the language is obligated and empowered to denounce it.
Language is the scaffolding on which we build all our knowledge and understandings of the world. It inherently informs the unquestioned truths that make up our worldview. As such, I believe it is the most important tool in any quest for social change. ( Which is also why I love literature so much!)