I don’t think it can be simpler then that. Unisex bathrooms allow anybody in. Man/Women bathroom allow man or woman. Until we have imaginary gender bathrooms, it’s super easy to go to the one your chromosomes or parts indicate. It does not subtract anything from your gender identity, just like going to unisex bathroom doesn’t make me gender-less.
Let me give you an example, you might be professional race car driver. But sometime for health reason you need to walk or pedal a bike. You cannot take your bicycle to track, or your race car to the park with moms and strollers. Yes it might inconvenience you, but there are reasons you are not allowed to do it. It’s called… other people and sense.