It is definitely complex. The point is that when you make a distinction, you have to justify why that distinction is useful and meaningful.

If a college student who's been on a team for ages finds out she has XY, then can she stay? What about a trans man on testosterone? What about an XX cis man? What about an intersex person?

And yes, mismatched hormones very much matter because they have biological effects. Largely the entire impact of gonads is their effect on hormones, and nothing else matters outside of the bedroom.

Like sure, in one context, female and male are referring to fertile people with particular organs. But why is that useful to society as a whole. Clearly it breaks down in a ton of cases. Why do we need a government document to record a vague estimate of reproductive ability?