I’m a transmasculine neuroscientist. Your argument suffers from the underlying premise that there is a unitary phenomenon of “biological sex.” In reality, most academics who undertake serious study of sexual differentiation, development, and behavior agree that “biological sex" can be assessed by multiple metrics. These include chromosomal sex, gonadal sex (ovaries, testes, etc.), morphological sex (the appearance of the external genitalia), hormonal sex (which sex steroids are present in that animal in which amounts), and behavioral sex (whether the animal displays dimorphic behaviors associated with males or females of that species). In people, sex is assigned on the basis of morphological appearance at birth. Sexually dimorphic behavior in humans is profoundly influenced by social/cultural factors, and much of what we would term “behavioral sex" in other animals maps onto our human constructs of gender identity and gender presentation. (This information is not new and can be found in undergraduate textbooks of behavioral endocrinology.)
It is unclear to me why morphological sex should take precedence over all other measures in assigning a “biological sex" to human beings. We typically do not have a method of gauging a person’s karyotype or hormonal levels. A person’s external genitalia are neither visible nor relevant to routine social encounters in most human societies. It seems deeply phallocentric to me that gender-critical philosophers use the presumed presence of a phallus as a litmus test for biological sex, rather than aspects of behavioral sex. Frankly, this stance is also deeply transmisogynistic, as trans men who alter their hormonal profile will adopt many male secondary sexual characteristics, but trans women who alter their hormonal profile post-puberty cannot reverse secondary changes like voice drop and facial masculinization (the very characteristics that are commonly used to assume whether a person has a phallus).