I was an American in Ecuador at one point. I don’t think being asked to give some insight as to why you look the way you do is a matter to get offended about, but it really is a polarizing topic. Some prefer not to discuss how they lost their leg, and others are more open to telling the story.
My point is, if you’re in a context where you seem out of the ordinary — in absolutely any way possible — people will take notice. It comes down to how you view their notice, and there is no wrong or right way to do so. Do you feel like an outsider as a 6'4" white American in an Ecuadorian marketplace with predominantly indigenous women averaging 5'4", or do you try to laugh at the ridiculousness of it all?
The concept of race won’t ever go away, as long as there are even micro-shades of difference between your skin and mine. The trick is to accept those differences and then continue past them.