I've done this in the past. It's not just annoying on the inviter, but it's also guilt-provoking for the person canceling.

I've learned that I usually know immediately whether I'll be up to accepting an invite, and if not, saying no immediately is better for all concerned.