This is a great point, and I’d like to add that racism is a huge social problem not because of individual racists, but because of the racist power structures in which everyone operates. If I don’t like people with blue eyes, I’m not oppressing them. Even if I got together with some weird friends and we were mean to everyone we saw with blue eyes, blue-eyed people wouldn’t be oppressed. They’d be harassed, sure; and we’d be jerks. But to really get some discrimination going, we’d need power, particularly the power to define “blue-eyed” as a separate, lower social class.
Elites have that power; some dude in a Duck Dynasty shirt does not. (He benefits from it, but that’s a separate discussion. Also, the actual Duck Dynasty guys are a different story: very wealthy, well-connected, strong media presence, very image-conscious and politically involved.)