Please read the whole thing, because I do make a strong case for your argument towards the end (yet not how you frame it).
Is it moral to smoke or sell cigarettes? No, because of the aggregate damage it produces. However, it is moral for one to guarantee a person’s autonomy, which allows them to buy and sell cigarettes.
Is buying or selling sex moral? This is a clear no for you, but even if it is a no, you are still left with needing to respect a person’s autonomy in order to guarantee a moral society.
However, I don’t even think conceding the first point, of whether buying/selling sex is moral, is valid. Is a man/woman immoral for sleeping with many different partners? I would say clearly no.
Does the exchange of money automatically mean the buyer is exploiting the seller? No, if the exchange of money meant automatic exploitation, then every person selling something should be considered exploited.
However, are people who feel forced into a profession exploited? Yes, and this happens across many different professions. The immorality of forcing someone into a profession is especially egregious when the cost to the individual is high. I would say that sex work has a high physical and mental cost.
Forcing someone into a profession can be explicit or implicit, by not giving reasonable alternatives or by not guaranteeing an income despite employment.
In our current system, I would say that the vast majority of sex workers are exploited because they are explicitly/implicitly coerced into a profession, which has a high physical/emotional cost. This means that the vast majority of men, who pay for sex, are acting immorally.
As I have illustrated above, the inherent act of sex can be considered moral and the inherent act of commerce can be considered moral with morality not fundamentally changed when one combines the two. However, given the externalities of explicit/implicit coercion, buying/selling sex can be considered immoral.
This analogous to any service/product, like the selling of Nike sneakers that is produced by child labor.
Therefore, if one wants to expel the immorality out of sex work, one should provide viable alternatives or guaranteed income, because the inherent act of sex and commerce is morally good.