That relative discomfort in socializing with a woman as opposed to a man at a party does seem rather harmless at first, though it can grow to be more problematic as this selection towards likeness moves further into areas like acknowledging expertise and successes, hiring, giving raises and promotions… The more one group sees a different group of people as “not like me, not one of ‘us,’ not part of my world,” you can see where that will cause problems in any setting where money and teamwork are involved.
While feeling uncomfortable around people who are different than you is a very human, very normal experience, I don’t think that makes it defensible once it has been shown to reduce the effectiveness of yourself or your team. The thing that makes humanity so great is that we can learn and adapt, anyway. It may seem more uncomfortable at first, but if you can, getting past that discomfort leads to bigger networks, more productive teams, a wider range of knowledge and expertise, and more. This works not just between phenotypically diverse groups, too, but for different professions and lines of work as well. The mixing of a diverse range of disciplines and backgrounds leads to the generation of all sorts of interesting ideas and solutions.