Good point, I agree.
I also feel that the amount of people I would be saving would surely impact my decision.
That being said, I would still want to consider if I would want to be pushed if I was the fat man. I.e. would I sacrifice myself for 5 others? Probably not. For 5 million? Yes, most likely. Therefore, it would be easier for me to push the fat man if there were 5 million people I would be saving, but not if it was only 5.
In coming to this conclusion, I am also only speaking to my own morality. Of course, then we can talk about arriving at a number that would be a tipping point in changing my decision.