Refusing to date a transgender person because they are trans is gender-identity discrimination, by definition.
Our choices are discriminatory when we make them on the basis of perceived distinctions between things. My choice to eat this red Skittle out of a bag of a hundred different Skittles isn’t necessarily discriminatory—I might have simply reached my hand into the bag without looking and removed one at random. But if I choose not to eat the yellow Skittle and instead choose to eat the red Skittle because it’s red rather than yellow, orange, green, etc., then I’ve discriminated because I’ve distinguished according to color and acted accordingly, based on some value system (red tastes better than yellow, or whatever). We do this sort of thing all the time, and we can call it a benign act of discrimination.
Discrimination starts to become malign when we make our choices on the basis of, say, someone’s race, gender, sex, religion, national origin, health status, etc. At least, these are some of the categories our society has chosen to protect for certain purposes.
None of this undermines your core point. Choosing a sexual or romantic partner is fundamentally different than choosing a candidate for a job, choosing a renter for an apartment, choosing customers at a restaurant, etc. The former is considered to be strictly private, while the latter are all subject to some degree of public regulation.
But that’s not to say that our dating lives and preferences are value neutral or immune from scrutiny (at least our own scrutiny, even if we want to avoid the policing of others). There are three questions involved here: (1) What choice are you making? (2) Why are you making it? (3) What should you think about it? (Is it good/bad, how does it effect people, should you engage in some thought about it, etc?) It seems like you might be neglecting #2 in your analysis, simply leaping from “I’m choosing X” to “And that’s ok because it’s my right.”
If I date only non-transgender individuals, I might or might not be transphobic. Of course, “transphobia” is just a crass label for fear, anxiety, or dislike of trans people as such, and it’s generally far more useful as a weapon than a tool for productive discourse. But regardless of the label, anxiety about transgender people, even subtly, might be fueling my decision not to date trans people, and that seems important to weed out. Though it’s also possible that I simply haven’t encountered a transgender person who seems like a “match,” all things considered. Whatever the case might be, it seems worthy of reflection, not knee-jerk reaction. It’s not enough to say that my dating choices are private or that a trans person has no right to be considered attractive, etc. Sure, that’s all true. But have some self-reflection.