The main problem when talking about sustainable energy systems is always economics. If we’re serious about transitioning to a low carbon industry, we need to fix, as you say, the economics of base load.
We don’t have many options for the base load. Either we make large storage a good investment, or we use nuclear, or we keep fossil fuels. A strong carbon tax would rule out the later, while making storage a more attractive option.
AI can help you predict those high wind events, but it won’t help you put in place the regulation that will push the markets to invest in the right areas.