As a man, I think I can answer your question about why the withdrawal, and I’m not afraid to talk about it with you since I am also a man with fluid gender.
My view is that sexual intimacy, as opposed to conquest [the chasers], requires so much vulnerability that in sexual intimacy we literally feel the most unsafe we can be. Our genitals are the most vulnerable part of us- unarmored and exposed, and they are also the conduit via which we have our most intense experiences of another person. On top of that, unless a man is a conqueror [sexually], he will be intimately connected to anyone he has sex with forever. An emotional doorway remains open. I think this is true for women as well, and across all sexual orientations.
The setup for a boy is that he discovers his sexual parts and is then instructed by lots of sources about guarding his being. The only safe route forward with respect to how to safely connect our private parts to others is when that other is a natural woman. It is gradually being understood pretty widely that for gay and lesbian people, this same natural partner will be the same sex, and parents and schools and friends will make room for this.
This understanding has not been developed for trans and genderfluid people, so cis men generally will feel that becoming too close to a trans woman is entering an unsafe and deep unknown.
Why not put themselves at risk for the sake of a good relationship? Take a look at The Mask You Live In -http://www.imdb.com/title/tt3983674/reviews?ref_=tt_urv . It details the extent to which men are trained to fear violence from other men if they lose their status as real men who dominate women and deviant men. Men are overwhelmingly certain they could lose it all- [their social position AND their sense of safe sexual identity]
The current societal ‘support’ for transgender everything is opening doors, so it is helpful if everyone who can will stand up and be counted. It sounds like the call from within you to come out is challenged by the reality that your mate will have to be someone who understands and embraces gender fluidity, so they are not afraid.
People grow- so call Jim- maybe he is ready to remember how special it was as well.