Yep, this. OP outlines behavior precisely, but I feel missed the root cause by a mile. Yours is much closer. I never feel shame for being attracted (well except when I am already in the relationship and then yeah, other woman being attractive for me, make me feel a little ashamed), but I exhibit behavior she is talking about. Your explanation makes much more sense to me.