Madison may have been entitled to that opinion, and he most certainly has a strong point, but Alexander Hamilton specifically maintains that the Electoral College will prevent the unsavory types from ascending to the office of the President, as the successful candidate must earn the support of the all the states, and therefore all the electors in Federalist Paper 68.
So, the contention that arguments were later invented to justify the necessity of the Electoral College are dubious at best.
The founders were no doubt arguing about a multitiude of reasons to avoid electing by popular vote. Slavery was certainly one of them. To assert that slavery is the preeminent or primary one cannot be substantiated. Such a claim remains merely an opinion.
Like this opinion: If we elected the President by popular vote, Taylor Swift would be starting on January 20. She would be writing songs about the former boyfriends who didn’t vote for her.