It's interesting Canadians think of barring condo development (and thus liniting the housing stock to sfh) as increasing speculation. Because of our housing bust in '08, developer speculation and condo construction go hand in hand in the American mindset. Then again we don't have nearly the same problem as y'all do with our housing being used as a speculative asset class for foreign buyers and assignment clause nonsense.
Zoning is an issue here oft brought up by Repubs as curbing development. It's tough because our city cores need lots of infrastructure development to handle big increases in density (heck, even to handle current density). So crazy you have so many sfh in your downtown. That must be nice for those who own there!