People who see everything in terms of race are racist.
Did you marry your husband because he was white or in spite of the fact that he was white? However you answer, you need to apologize to your husband for your attitudes.
I see similar sexist issues you need to deal with. I see no problem with single sex events, because quite frankly, there are some things men and women should only discuss amongst themselves, because the other sex will not understand. However, excluding the sex because you want to blame them for all the ills of society is not good.
If your husband took a position, solely becuase he thought he could win an argument, it makes me wonder if you ever let him have the last word on anything. Do you always have to be right?
