That was actually my point. Jews tend to look like the natives. Historically, despite carrying facial features similar to the natives in “country X”, Jews tend to be viewed as outsiders.
I was using Jews as an example to underscore that it is not similar appearance alone that breeds an more natural commonality between strangers. Rather, it is the assumption/expectation that somebody who looks like me thinks and acts like me. Because people did not traditionally look at Jews as sharing their culture, the similarity in appearance was meaningless.