They might be upset because it’s non-sequitor, or because it’s implied that it’s pejorative. They are certainly in dereliction of the public duty if they are slacking, but the color comment is simply cringeworthy.
If I were in a room with my Mom’s relatives and someone referred to us all by color or stereotype I would raise an eyebrow, same as if I were with some of my father’s relatives. You don’t call me red, or any other color. I wouldn’t refer to your race, for just about any reason. It just doesn’t seem necessary?