Let’s get this out of the way first: White guy, black wife, “black” children.
It’s all relative. My wife is African. Directly translating from her tribe’s language, “brown” is used to describe someone with light skin. I’m not brown, though, I’m a muzungu (literally “white person” in Swahili, used throughout most of East Africa). One of my wife’s cousins is half-white and gets called “blondie” by relatives (she’s not blonde). My (late twenties) daughter doesn’t think of herself as white or black, she is herself. My (early twenties) son hasn’t sorted out what he thinks yet. Would they get told “you’re not white” in a gathering in my wife’s country? Don’t know. My son was called muzungu when he did some work for another cousin, so probably not.