You’re right. And, for the most part, child-free women aren’t immediately suspected of being a predator. A non-father who likes kids, esp. little kids, is automatically suspect in a really ugly way. It directly contributes to the lack of men in positions of child ed, like pre-and elementary school teachers. Kids could use good men around starting at age 0, and suspecting child-free men of being automatically scary is really hurting society. (And it hurts a bunch of good men, too).
I think the blowback if you’re a woman can be more harsh (men aren’t generally accused of being not-men if they don’t have kids, or of being monstrous men, while men often get the ‘you’re not a real woman…’ or a wrong — monstrous — woman). But both negative reactions are really rough on those who have to deal with them.