This is a really interesting take on things. I recently dated an ex-Mormon man and the stuff he had about sexuality eventually broke us up. I’ve never seen such extreme shame in a man for having basically normal sexual responses.
One thing that happened recently is that I gained about 15 lbs or so and I was complaining to my then-BF that I felt fat and ugly. He got kind of peeved at me and said “If you’re fat and I’m attracted to you, what does that make me?” or something along those lines. He’s normally very calm and collected (repressed) and he actually got visibly bothered that I was expressing feeling unattractive. It was surprising, but seems to support your hypothesis. By the same token, I dated a large man one time and felt very self-conscious about it. In fact, most of the men I’ve dated have been pretty average looking (as am I), and I’ve been told that I ‘date freaks’. It’s odd to me that the physical attractiveness of our partners is then assumed to reflect on ourselves. If a man is attracted to an “average” looking woman, he’s “average” — and if an average-looking woman dates normal-looking men, she’s “settling”.