I certainly don’t think skin color is the only factor that impacts privilege. The idea of people having different starting lines is a is a much broader concept that applies even in countries where skin color may play a smaller role. Ethnicity, gender, sexual orientation, religion, and other other factors all have varying levels of impact as they relate to privilege around the world.
That said, for historical reasons skin color plays a bigger role in some countries than others (e.g. USA, South Africa). It is by no means the only factor, but it is a big one. My personal privilege had a lot to do with my gender, sexual orientation, and even the region of the US I grew up in—but it would be foolish to leave race out of the privilege equation in America.