Backtesting to the Future

If you built a machine that predicted the future, how would you test whether it worked? Ideally, you’d be able to get in your time machine and see whether your predictions were right. Sadly, our team, while brilliant, includes no Marty McFlys or Doc Browns. At TellusLabs, when we evaluate the strength of our predictive yield models, we rely on less dramatic (though still exciting) techniques.

We look to backtesting.

Our data science team uses more than 15 years of historical data to build our models. To assess their performance, they then exclude certain years to see how the models would have done through different periods of time in comparison to the historical record (or other forecast models). This shows how our customers might have benefitted and it also shows off the strength and solidity of our different models compared to alternatives.

For example: We tell our models to look back at weather, imagery and other variables from 2008–2016 for Brazilian and Argentinian Corn to understand their relationship to crop yield. Not only do our models come back with forecasts for this year’s yield, they also show how well they would have done vs. historical record. The chart below shows the average % error of our Brasil Soy model (in green) during the Jan-Apr period over that 10-year period, versus the government’s own monthly estimates (in black). You can see that, particularly early in the year, our models have much lower errors than the govt.

Backtesting is used all the time in finance. It’d be pretty cool if someone had a model to predict the price of bitcoin. Next time someone claims they do, ask them for the backtesting to see how well it would’ve predicted this year!

While we want our models to be accurate, we also make sure not to overfit our models. Overfitting occurs when analysts go overboard adding predictors to a model that may improve backtesting scores, but also introduces excessive features to the model (which could hurt its “out of sample” performance in the future).

For example, if you found out that including a variable for the sky’s blueness improved the backtesting of your bitcoin model, that wouldn’t necessarily make it a better model (correlation is not causation)! At TellusLabs, we make sure to strike a balance with our models so they not only backtest well, but also have variable relationships that make sense. Our strategies for avoiding overfitting are the subject of another blog post!

Our disciplined approach means that we are particularly pleased when we see great “out of sample” performance. Our Kernel models predicted the 2017 and 2016 US corn yields within 1 percent of the official year-end figure months ahead of the announcement by the U.S. Department of Agriculture (USDA).

What does that mean our clients?

It means strong, solid and high-quality insights to support you as you make important agricultural and economic decisions.

Email business@telluslabs.com for backtesting summaries!