This might be because, as Zizek and others maintain, the term masks oppression. If one is able to tolerate a person or group, then presumably one could also choose to not tolerate them. In other words, one tolerates a presumed inferior.
Tolerance means “the ability or willingness to tolerate something, in particular the existence of opinions or behavior that one does not necessarily agree with.” Synonyms are: acceptance; forbearance; liberality; liberalism; patience; charity; indulgence; understanding.
Intolerance means “unwillingness to accept views, beliefs, or behavior that differ from one’s own.” Synonyms include: bigotry; parochialism; prejudice; bias; partisanship; partiality; discrimination; injustice; inequality.
How can tolerance mask oppression? Tolerance requires us to recognize each and every member of humanity as an individual and affirm “their ability to improve their lives through the use of reason and ingenuity.” While it may not be sufficient to guarantee equality, it is necessary.
Tolerance is not “a kindly pat on the head from you betters.” It is solid recognition that we are all independent actors with the right to hold our own beliefs and choose our own path through life. It is recognition that we do not have the right to impose our beliefs on others. This holds for the majority imposing itself on the minority and for the minority imposing itself on the majority.
In short, our right to be independent actors, hold our own beliefs, and choose our own actions is limited once it begins to interfere or violate the rights of one of our fellow humans. Tolerance absolutely requires the recognition of the rights of others.
This is regardless of who is powerful. Tolerance works both ways. Power only enters the equation when there is a lack of tolerance. In the absence of tolerance, the powerful can force their ideas, their beliefs, and their rules on whomever they please. They can refuse to recognize those they disagree with as independent actors. They can refuse to recognize the rights of the less powerful to choose their own actions.
In other words, tolerance is critical because it serves as a constraint on power. Tolerance doesn’t mask oppression, it inhibits it.