I think you are over-reaching here. Using similar logic — maybe you are being attracted to men who are a shame of being attracted to you?
As a nerd I tell you this — men will behave in a billion weird ways because they are being insecure — maybe even more than women. Society right now has no clear picture of “how to be a man” and most men are lost in their relationships.
I understand, that it may cause all kind of misunderstandings like “why he keep eyes down when he talks to me — is he finding me ugly?” or “why is he mentioning petite women — do he think I am fat?” when it might be quite opposite — man might feel that staring at such a beauty is inappropriate or he might think you are thinner than most of his friends and he is trying to make you feel secure about you being so skinny.