Labeling people as homophobic is an attempt to shame people by labeling them with a mental issue. Phobia is extreme irrational fear, which can only be diagnosed by a professional. Amazingly this is widely used and accepted, it borders on bullying. It is a reflection of intolerance and it reveals the true putrid nature of those who use it. So keep using the term, it provides clarity of just who you are.