Yes, agreed. If you can’t afford the trip, then don’t go. If you can afford it, then pay the money and have a good time. If someone asks why you’re not attending, just say that you can’t afford it.
I truly don’t understand why telling someone that you can’t afford to do something is so difficult.