You raise some good points.
It’s true that the idea of money being speech is held up by court cases like Citizens United and Buckley v. Valeo. Those did rule that campaign finance is protected by the first amendment. But again, I would argue that this is “speech” that is directly hurting others, which isn’t supposed to be…