The feminist belief (I consider it so not because of any feminist writing I am aware of but because it is a belief common among feminists) that males should not have/do not deserve any privacy or any time away from women is unfriendly, unhelpful, sexist misandry.
Feminists, both male and female, would not like a man to walk into a female locker room while women are changing. (They might just have to put up with it soon, given that Obama’s declaration can lead to anyone who momentarily identifies as female will be legally entitled to be in the female bathrooms.) If or when men do enter women’s spaces, they will discover not just playful banter but outright misandry and, at times, hateful plotting. More than one family breakdown has been engineered in a female-only space: something most men would never tolerate even if they had any male-only spaces left to them.
Men are men, not women. That is now the vast majority of women want them. There is no ‘toxic masculinity’ but there are toxic people and there is toxic human behaviour and attitude which can be found in both male and female. So long as men are just having banter and joshing one a other, sure some feelings can get hurt but no-one comes to any harm. Harm comes in continuing to limit men’s human rights but also constrain their personal and private interactions.