Well you are entitled to use any word I use. Are you not?
Adam Koncz

I don’t wish to speak on behalf of the person your question was directed to but if you are asking that question it might be worth you thinking about a few things:
1. I think the racism and structural prejudices in our society mean that many black people and other people of colour are not able to say everything that we, as white people can. Take the most extreme example- i believe if i was pulled over by a cop in America for no reason I would feel reasonably comfortable, or at least not in danger, in questioning why they had pulled me over. I think if I was a person of colour, fear (and a very justified fear) would prevent me from speaking. So, no, black people cannot say everything that you can. 
2. The way our society is structured often hides the words of black people. Their voices are made less visible than white voices. This means that when they may able to say every word you can, their words can be unjustly robbed of power and influence. For example, ‘mainstream media’ (by which i mean media most readily available to most people — e.g.whats on at the cinema, on the TV) underrepresents black people.
3. Obviously, just to reiterate this point, as a white person you benefit daily from the oppression of black people — this does not mean that your life is automatically better than every black person everywhere or that you don’t experience any other forms of oppression or that any suffering you experience is less valid — but you do benefit. You may not want to benefit but you do. This means that in your mouth the n-word is offensive because it shows a lack of respect and understanding of that fact. A black person does benefit from the oppression of black people — therefore the n-word in their mouths may not carry such an offensive meaning. 
4. I imagine that in situations like formal jobs etc. black people are also not allowed to use the n-word. There has definitely never been a law which stipulates it is illegal for white people to say it. Its just so much worse when we say it.

Also you said this: “ How would that work the other way around? Nope, non-whites cannot say these words or wear these clothes (for reasons of cultural appropriation) unless you share the “white experience”? Wouldn’t that be the most nazi thing ever?” — I think you are almost proving the opposite point here — because to try and make the idea of people dictating people’s words/clothes etc. on the basis of skin colour sound bad and ominous you are forced to switch the example to one of white people doing this. That example is so immediately distasteful (or as you put it ‘the most nazi thing ever’) because the power balance is so obviously in favour of white people. It isn’t arbitrarily about skin colour — the n-word is about skin colour and therefore it is appropriate that the social ‘rules’ around that word also be defined by skin-colour. 
Its about your understanding of what the term racism means. It does not simply mean differences on the basis of skin colour. That is probably something more aptly described as racial prejudice or something more neutral. To view racism in such a way is to ignore the fact that racism as we know it today was something purposefully created by white people in order to exploit other people. It is so much more entrenched and complex. This is why calls of ‘reverse racism’ are often unfounded. I would say most cases where white may feel like they are being treated unfairly because of their whiteness it will be actually be that:
a. They are being treated unfairly, but because they are poor not because they are white.
b. They are to self-entitled and out of touch with the world to understand the huge difference between not being able to say a word that other people can and having to deal with actual racism day in and day out.
c. They are insecure and feel that being told you have privilege is an insult or they feel that being a victim of racism gives you some sort of advantage or social currency in the world that they want.