I agree with this, except for one part: “you can always refactor.”
You should have turned around to your boss and said: I can deliver this in 2 days in front of customers (giving fast turn around on solving the urgent business concern), but I’m going to need 2 weeks to do it properly afterwards (plugging the equally critical engineering concern).
Hampering the cost of change of a software project is just a problematic to an organization.
If it hurts, do it more often. So refactor son!