Lee did what he could in opposition to slavery and secession, but in the end the establishment overrode him. It’s easy from another viewpoint, without experience of the context and the time.
In the end it was a choice between defending his home and family from a brutal invasion of the Union army, the very army he served with earlier. Lee could not stop the devastating burning of Richmond and his neighbor’s homes, nor the barbaric scorched earth march of Sherman through the rest of the south.
But after his surrender to Grant at Appomatox, he was proud to rejoin the Union in conciliation and unity. He welcomed the freed slaves into his home and his church, and urged everyone toward a peaceful new order.
No historical figure of any significance was perfect. Even the revered Lincoln allowed the non-secessionist states to retain their slaves during the war, until the 13th was passed, and his outspoken solution to the freeing of slaves was to ship them back to Africa. It’s no secret that he frequently sent journalists to prison for speaking out against his policies.
If we are to get rid of every monument or memorial to every historical figure that was ever on the “wrong” or losing side of a battle or a conflict, I can’t think of any we would have left around.