But the point is that having a Y chromosome or having certain genes does affect how your brain develops and thus affects your cognitive functions.
Citation needed much? This assertion is extremely flawed and founded upon no scientifically sound evidence. The whole point here is that brain development and how it correlates to abilities is almost definitely a result of socialization, especially given that neuroscience has been unable to demonstrate such a link, as you assert, while psychology has long been proving over and over and over again that these differences we once attributed to biology are, in fact, the result of socialization.
So no, you’ve got it backwards: the thing you’re asserting is the ideology, and the response against it is the science.