I completely agree a lot of men are abusive idiots, but you’re pointing to the larger divide between people who deal with harassment: some, like your boyfriend, try to ignore it, and others engage in order to deflect it. I’ve always been of the engagement school: on the street, for example, give a friendly smile and, if possible, keep moving. The people like your boyfriend who ignore harassers end up denying their humanity, and for many of the people whose desperation manifests as harassment, that only enrages them.
I don’t mean to suggest there’s one solution for every situation. Still, when in doubt, I prefer yours.