(meta note: is feedback of this general quality level useful/causing any new thoughts?)
> transfer the learned models to a new domain in which forecasts aren’t available.
What I think I’m proposing is something like a toy model in which only one domain is “blinded.” Your training data is across several domains and the inputs to the model are the parameters that the experts consider useful for selecting a prediction algorithm for the domain (or data that proves this?). Discover some sort of fit from prediction algorithm parameter->problem domain parameter that predicts making tighter CI predictions (that turn out well calibrated) than a control of applying random prediction algorithms to the domain. Once you have some plausible outputs you feed it data from the blinded domain and see how it does.