Seeing Around Corners: Complexity, Tweets, and Trade Wars in the Future of Commodity Price Forecasting

Jul 2 · 5 min read

The modern global oil market is extraordinarily complex. Collectively, we consume more than 30 billion barrels of it each year. Millions of workers operate trillions worth of equipment to pump it directly from the ground, deep beneath the oceans, or squeeze it from shale rocks. Next, this liquid gets sent to hundreds of thousands of refineries dotted all over the planet, who cook the raw material until it separates into gasoline, jet fuel, kerosene, and other everyday energy products with which we are all familiar. Finally thousands more vehicles carry these finished goods near to us for convenient purchase and consumption. And that’s just the physical side of the market.

Financing all this activity is a pool of global liquidity concentrated around the major commodity exchanges in Chicago, New York, London, Shanghai and Singapore. Speculators take the other side of the physical market, shouldering price risk for producers and refiners seeking to hedge their exposures. These traders carefully monitor market conditions to form expectations about the current and future state of global supply and demand for oil and its derivative products. Millions of contracts trade daily, transferring trillions of risk annually. All of this physical and financial activity is coordinated via transparent prices, which aggregate and crystallize the enormous quantities of data generated by the billions of people worldwide who participate in this market — either in producing or consuming, buying or selling.

Why is forecasting prices so challenging?

This is why forecasting commodity prices is so challenging, confounding generations of mathematicians and economists; the sheer scale and complexity of the market is literally mind-boggling. No human could possibly forecast where prices are heading accurately. Despite the difficulties, some of our best and brightest social scientists have tried (and failed). Paul Samuelson, Nobel Prize winner and the “Father of Modern Economics”, demonstrated in 1965 that under a rigid set of assumptions commodities follow a random walk. It took nearly 30 years of research until another Nobel Prize winner, Angus Deaton, revealed that much “randomness” is actually driven by the impossibility of negative storage. Because the physical market must clear, prices can swing wildly and in seemingly unexpected directions.

Today, it is well established that the bulk of all commodity price movements are driven by a single global factor, which we can call the international business cycle. It was nigh on impossible in the 1960s to forecast prices, and the global economy is bigger, more integrated, and more complex than ever. Improving on forecasting at this stage is therefore extraordinarily challenging. The number of variables that can influence oil prices has exploded after decades of globalization and financialization. Everything from the President’s mood to social unrest in Venezuela, to military confrontations in the Middle East, to monetary policy decisions by the Reserve Bank of Canada, to industrial policy choices made by the Chinese Politburo, to the weather in Europe, to credit conditions and earnings reports in the United States (and more!) can and do move international oil prices.

Can Machine Learning and Alternative Data help?

To explain these price movements, huge stores of data from a range of diverse sources are required, and can be combined in countless ways. This is a problem machine learning is well suited to address, and why “alternative data” is finding its greatest utility in commodity markets. The world is actually covered with useful information, and more devices than ever are capturing it. Satellite imagery, once rare, is now widely available. Social media too generates vast troves of data. For example, personally I first learned of Russia’s invasion of Crimea back in 2014 while sitting on an oil trading desk. A local Ukrainian noted the presence of tanks crossing from next door and posted a photo on Twitter. It took a few days before everyone realised what was happening and market prices began to adjust appropriately.

Seeing around corners with AI

The challenge isn’t just getting your hands on useful data however; it’s knowing what to do with it and what is available. To illustrate, researchers at MIT are using deep learning and image recognition techniques to literally see around corners. But actually, the information needed to see around corners has in fact always existed, hiding in plain sight. It just wasn’t possible to make that information useful until now. Where our eyes only see vague shadows projected on the ground amidst dappled light, a carefully trained machine learning algorithm can precisely reconstruct an image of the scene behind a physical barrier, solely from the way visible light is distorted by those hidden objects.

If seeing around corners sounds like a superpower, that’s because it sort of is. And in much the same way, here at ChAI we are applying those superpowers to international commodity markets, beginning with the king of all physical commodities, crude oil. The information required to improve forecasts has likewise always existed. Ships that are full of oil are heavier, and thus sit lower in the water than ships that are empty; refineries running at full capacity produce brighter and longer burning flares than those that aren’t. The problem was that it was just too expensive to collect and even more expensive to sift through. Welcome to the future of commodity forecasting. Signals might arrive via a tweet, or a trade war, and artificially intelligent agents are here to help you anticipate those twists and turns.

Author: Ben Falk

Originally published at on July 2, 2019.


Written by



Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade