Just because times during which world temperatures have been cooler by 1 or 2 degrees C have been hard times doesn’t mean that 2 or 4 or 6 degrees warmer will be an improvement. There’s a sweet spot somewhere, after all.
We know from past experience what happens to crop yields during heat waves. The graph of yield against temperature looks like the first bit of a roller coaster: as temperatures progress from cooler to normal yield slowly rises. As they progress a tad beyond normal, yield rises more slowly. And as they progress beyond that to just plain hot, yield collapses.
The temperatures we must expect from all this extra CO2 will involve many more heat waves than we used to have, and thus many more years of crop failure.