Some really good points here, but I want to add some thoughts on the caveat about trust. The importance of trust depends a lot on how the model is used. If it’s part of an automated business process, then even a small lift should improve the business process, and yes, deploy. But if it’s part of a manual business process, where a person has to see the prediction and figure out how to incorporate it into decision-making, a weak prediction can be much worse than, say, a graph of contextually-relevant historical data, or in some cases, than literally no information at all. The very last thing you want to create in consumers of a predictive model is the thought “I’ve been doing this for years and this stupid computer doesn’t know what it’s doing.” Bad first impressions are killer. Be humble.

One thing that can be helpful, if your model supports it, is some sort of posterior predictive interval. If your model can say, “given the information available, the temperature tomorrow will likely be between 56 and 93 degrees,” the user will properly discount this information, especially if sometimes the model is very specific. Generally, if the model has some reason to believe the prediction is inaccurate, and trust is an issue, you’re better off replacing the spurious prediction with a ¯\_(ツ)_/¯.