Yep this is me as well. I have as little to do with strange women as possible. I interact with women regularly at work, but never alone, never after work, and never in anything but a cold professional manner. I would rather be accused of being a robot (which I have been called) rather than accused of sexual harassment. Sexual harassment, sexual assault and rape are all intertwined in the minds of many people. So if a woman takes offense that you had the gall to ask her out for drinks, she claims sexual harassment, then many will assume that you are just as bad as a rapist.
It has always been my rule to never date a co-worker. Because I work a lot, it has limited me socially at times but I have never had an issue of a woman accusing me either.