Not sure chatting with a hobo for ten minutes is necessary to you and your boyfriend’s safety. I agree with him that you dragged that out and made it painful. The title says smile at men, not dedicate your time to them. Say “thanks, have a great day!” and walk off, or say “screw you buddy!” and walk off, or just walk off (really that walking off is the key). You don’t hang out and make friends with the catcaller. Best case scenario you lose your own time and piss your company off. Worst case scenario you lead the catcaller on and he does something desperate when you finally work up the nerve to mention that you have no interest in a chat.