First of all, I applaud your decision to travel with your husband. That is a very wise decision thing to do.
Second of all, you sound hurt by the negative opinions that some people have on your decision to spend money. I am sorry to hear this.
I can tell from experience that they might be envious of your decision to travel and they attack you now as a way to say that there is a consequence of your decision.
And it is ok.
You will bounce back from this.