What we need is a moral society. Not a perfect society, but a moral society.
Atheism has no record in that direction. No atheist philosophy has produced any difference between moral or immoral or amoral behavior. The only constraint that matters to the atheist is the police power of the society they live in.
An immoral atheist’s actions matter greatly to his or her victims, but not a whit to the atheist, as long as they don’t get caught.
Our slaveholders and their slaves are all long dead. Since they were simply atoms created in long dead stars, briefly animated by natural processes, and they briefly interacted, nothing that happened between them is of any consequence. Their atoms have since dispersed; those atoms are no longer animated. Nothing about them matters. Atheism is nihilism.