This is why many feel that cultural appropriation is not even a thing. Disrespect for people’s culture is one thing, but this idea of judging who appropriates what mainly based on color of skin or how much they know about the culture (something difficult to decipher btw) does seem backward.
Cultural appropriation is mostly an emotional appeal for defendants of former oppressors who destroyed and disdained foreign cultures to leave them alone, in part as punishment for what their ancestors did. The logic falls flat when you consider marginal cases of people of with upbringings that are different from what one would typically guess like a white person born and raised in Japan wearing a kimono, or other cases of experience such as A black American who is a scholar of African studies wearing a dashiki or some other tribal specific ornament or garment at Afro pink fest, etc.
So I believe we need to rethink this cultural appropriation thing and define what we really mean and what we expect in order to not feel like our cultures are being disrespected, keeping in mind that one Yoruba man cannot speak on behalf of all Yorubas for instance. This is where I agree with you that discussions must be had, but I’d be careful to make sure unity is maintained between ethnic groups of African and Non-African (geographically) Blacks.