What if you can’t walk away- you’re waiting for a bus stop, or a friend to come pick you up?
What if the catcaller follows you?
What if, incensed by your apparent disinterest, the catcaller becomes violent?
What if, when you “work up the nerve” to blow him off, as you say, he does not accept no for an answer?
I’m not saying that catcallers would always exhibit these kinds of behavior, but you can never know how they would react to being ignored or avoided. Engaging them in conversation, on the other hand, has a fairly predictable safe outcome.