Yeah, I totally see your point and I think there are many who use the term far too loosely as just a smug condescending way for one to use purely bias rhetoric against opposing viewpoints. However, I do argue that if someone is referring to a group or person who has been prove to come to a conclusion/s from illogical reasoning or have arguments saturated in fallacies then the phrase has good merit for it’s use.
If they use it to refer to anybody’s argument that has logical reasoning behind it and no major fallacies/flaws in their arguments then your response is absolutely correct. So it depends on the arguments being held under the light by someone and that solely determines if someone is being anti-intellectual about a subject.
If the same person constantly proves to argue in the same capacity about everything they believe then it is safe to say that they are an anti-intellectual. One faulty argument does not prove ones intellectual capacity overall.