It can be easy to forget that sexual assault is a crime committed by a very small percentage of the male population. You might think otherwise, considering how many women have come forward in the last year with tales of abuse and the epidemic number of sexual assault victims in the U.S. — one person every 98 seconds.

The truth, though, is not that most men abuse women — it’s that the small number who do are able to get away with it multiple times. For example, according to a 2002 study, six percent of men attempt or commit rape, but more than half of them will attack more than once, averaging about six rapes each.

Why is this minority of bad men able to get away with abusing women over and over? Because “good” men make it easier for them.

You don’t have to be an abuser to enable abuse, and over the last few weeks, Americans have watched that reality play out on the national stage.

Knee-jerk sympathy for men accused of wrongdoing isn’t new.

After CBS chairman Les Moonves was accused of sexual assault, for example, network board member Arnold Kopelson said, “I don’t care if 30 more women come forward and allege this kind of stuff… Les is our leader and it wouldn’t change my opinion of him.” Phyllis Golden-Gottlieb, who says Moonves raped her in 1986, told New Yorker reporter Ronan Farrow that she didn’t come forward for fear it would derail her career. Comments like Kopelson’s suggest she was likely correct.

When New York Review of Books editor Ian Buruma was asked why he would run an article by Jian Ghomeshi — the Canadian radio host accused by nearly 20 women of abuse ranging from sexual harassment to punching a woman in the head so hard she couldn’t see straight — he responded that “sexual behavior is a many-faceted business” and that “the exact nature of his behavior” wasn’t his concern. He was not the only man unconcerned with Ghomeshi’s behavior: when a CBC employee brought forward a sexual harassment complaint against him in 2010, she says she was summarily ignored.

And, of course, in the week since professor Christine Blasey Ford has come forward to accuse Brett Kavanaugh of attempting to rape her when they were in high school, we have heard from a parade of men (and some women) eager to defend the Supreme Court nominee.

Some say Dr. Blasey Ford must be lying; Mississippi Republican Senate nominee Chris McDaniel, for example, said that “these allegations, 99 percent of the time, are just absolutely fabricated.” (In truth, false allegations are exceedingly rare.) Others claim she must be mistaken or misremembering. In the Wall Street Journal, Lance Morrow writes that “the passage of time sometimes causes people to forget; sometimes it causes them to invent or embellish.” Besides, he says, “no sexual penetration occurred.”

Most disturbing are the men who suggest that pinning a girl down, covering her mouth, and attempting to rape her is simply normal teenage boy behavior. A lawyer “close to the White House” told POLITICO that “if somebody can be brought down by accusations like this, then you, me, every man certainly should be worried.”

Knee-jerk sympathy for men accused of wrongdoing — something feminist philosopher Kate Manne dubs “himpathy” — isn’t new. For decades, studies have shown that men are less likely than women to believe a woman when she says she was raped and are more likely than women to hold negative views of rape victims.

Even more dangerous, though, is the message that abusers receive: that they will be protected.

This isn’t just data: it’s a map to understanding how abusers are able to flourish. Consider the disproportionate number of men who are police officers, judges, magazine editors, movie directors, college administrators, and bosses. The people who shape how we think about gender and power and who decide how accusations are handled are far more likely to be men — who we know are more likely to empathize with male abusers than female victims.

This bias not only causes demonstrable harm — like HR complaints gone unanswered or reduced jail time for rapists — but it also has a profound psychological impact on both victims and assailants. For women, there’s a chilling effect on our desire to come forward. Why would we report abuse when we see that the consequences for victims are often far worse than those for our attackers?

Even more dangerous, though, is the message that abusers receive: that they will be protected. That they can do what they like because no one will believe their victims anyway. As feminist writer Thomas Millar put it nearly a decade ago, “It takes one rapist to commit a rape, but it takes a village to create an environment where it happens over and over.”

Until men who don’t abuse women are willing to grapple with how their biases pave the way for those who do, harassers, rapists, and abusers will continue going undetected and unpunished. As Senator Mazie Hirono put it this week, “I just want to say to the men of this country: Just shut up and step up. Do the right thing for a change.”