Is there some reason why you have to keep seeing this person or need to allow them to be near you?
It really sounds like you’re not getting much out of this “relationship” to me.
Are you afraid he will hurt you if you leave or refuse to see him anymore?
I am not trying to be mean, when we are in “relationships” like this we always have reasons. None of them are bad. You just need to sort this out and understand yourself more, so you can get strong enough to get away. It is hard, but I promise, being away is MUCH easier once you are really away, than what you are dealing with now.
I don’t mean to push you too hard, but I feel like you are here, talking openly about it… maybe you are looking for a push?
What do you think you need?