As a former carnie, I’ve got to imagine there is some way to make a portable stadium at least. Multiple banks of seating that fold up to fit on semi trailers, with a big framework of steel beams, that breaks down into similarly transportable components. As for a swimming pool, do they need much more than a big ditch with a plastic/rubber liner? The cost of building the portable Olympic Stadium (and later maintenance) could be shared by all participating nations. Then host cities would only have to pay for approved transportation and setup.

Obviously housing is a whole other issue.