Interestingly, I grew up in one of the very few families of colour in my hometown of Lincoln, England. Until the 70s (kenyan expulsions) I think we knew all of the other families of colour in Lincoln. I pass for white, but some of my siblings do not.
There was a lot of casual racism, a lot of ignorance, and a small but very real amount of quite vicious intentional racism. Before we moved into a middle class neighbourhood people on the street mounted mounted a letter campaign to try to prevent us from buying a house.
Once when we had kicked our football over the garden wall, I went with my dad to ask for it back. As we waited at the door, we heard the sound of the ball being bounced on the floor. Then it stopped, and we heard an unmistakable hiss, and the man opened the door and wordlessly dropped the flat ball in front of my father.
As I have white skin, I would often hear racial comments dropped casually intended for others — in fact I still do. Now I have the guts — and times have changed enough for me to to call people on it. Back then, it left me divided and angry, as I lacked the courage to stand up to it.