AI Augmented Commodity Price Forecasting

This article is the first in my series of articles aimed at ‘Commodity Price Forecasting’ which will cover the business aspects and need for commodity price forecasting.

Outline for this series: Some of the major areas which will be covered includes the following:

Part 1: Business context of commodity price forecasting

· Industry use case — Importance of commodity price forecasting

· Current pricing mechanism

· Need for AI/ML augmented price forecasting

· Structuring commodity price forecast problem

Part 2: Algorithm and some common issues in real world deployments

· Complete pipeline for structuring AI/ML based commodity price forecasting

· Challenges with short time series

· Out of sample distribution/data drift

· Model interpretability/explainability

Introduction

Commodity price forecasting tends to play a critical role in the strategic plan of procurement and pricing of products that rely on these commodities as raw materials. Forecasting empower the planners to make data-driven decisions, reduce pricing-related risks, and proactively manage suppliers while minimizing disruption in the supply chain because of price volatility. For manufacturing heavy companies, raw material bulk procurement can be upwards of 50% of cost of running the entire business. Hence, the need for accurate price forecasting for business to make more informed pricing decisions has become more important.

How Raw Material Price Forecasting Helps

Some of the areas where forecasting is a critical value proposition for planning and production in today’s modern manufacturing landscape are:

1. Purchasing raw materials: With no prior information on future order volume, industries often purchase raw materials without full knowledge to make sure that their factory is in position to produce the goods as soon as the order is placed. Hence, in these scenarios price forecasting becomes quite helpful in procuring the required raw material when the prices are low.

2. Sensing future demand: For few manufacturing industries, demand is highly tied to raw material prices, which helps in anticipating the future demand.

3. Pricing of finished good: Anticipation of raw material prices is a key to product pricing and market competitiveness. For semi-finished good industries, the final product price is usually some factor of raw material price.

4. Production planning: Better production planning comes from better demand sensing which in turn in is dependent on raw material price forecasting.

5. Managing inventory: An opportunity to buy raw material at a lower price is always balanced with existing inventory to not hold up cash.

Traditional Price Forecasting Mechanism

Many businesses rely either on analyses from business experts or on statistical models. Some of the ways in which business manual forecasting is based on are:

1. Market Reports: Reputed vendor such as IHS Market, S&P Global, ICIS, CRU, Baltic exchange etc.

2. Brokers: Manual data collection often by calling brokers or communicating via WhatsApp etc.

3. News Report: Unstructured news from news agencies, ad-hoc news sharing via WhatsApp etc.

4. News Aggregators: Bloomberg, Google etc.

5. Vendor Sensing: Sensing if vendors are calling us or if we are calling them to sense if price is moving up or down according to vendor.

6. Experience: Experience based logic related to port stocks, operational capacities of plants, oil prices etc.

With the above-mentioned ways, companies make educated guesses on what the future holds based on the expert’s expertise and previous decisions. Unfortunately, because of high volatility and complexity associated with the commodity prices, traditional (manual) way of forecasting mechanism sometimes fails to provide reasonable forecast. Hence, there is a need for an alternate price forecast which could augment the manual forecast and help business in taking more informed pricing decisions.

Need for AI based Forecasting

The traditional forecast methods are either based on quantitative and qualitative analysis of supply and demand side indicators or statistical models using univariate approaches relying exclusively on historical price data. However, these methods fail to capture the complete market dynamics and does not even perform well for longer time horizon forecast (weekly/monthly). Whereas AI/ML based forecasting models can provide more accurate forecast even for longer time horizon. These algorithms are also capable of processing large amount of historical data and find hidden patterns to help companies take better decisions. Some of the benefits of AI based forecasting are:

1. Capable of dealing with high price volatility

2. Capable of integrating multiple predictors from heterogeneous sources

3. Capable of providing accurate forecast for longer time horizons

4. Model Interpretability or understanding an impact of a variable

AI Augmented Price Forecasting

The below figure shows some of the examples of forecasting with different time horizons (lookahead) and challenges associated with them. For short term forecasting i.e. with time horizon as seconds/minutes/hours, the challenge is to deal with the higher computational complexity arising from the large historical data. Whereas, as we move to the right for longer time horizon (>= week) which is usually the case with commodity price forecast, the challenge is to deal with less historical data and large number of predictors as its easier to collect more indicators on a weekly and monthly basis. This results into a short time series problem with p>>N (no. of predictors > observations) which in turn requires a careful selection of algorithm to provide a reasonably accurate forecast. Due to the above-mentioned challenges with AI based forecasting for longer time horizon the AI based forecasts are not capable of completely replacing the existing manual forecast and are more suited for augmenting the manual forecast.

Fig .1. Time horizon based different types of forecasting

I will be covering the challenges associated with short time series problems and ways to mitigate those challenges in my next part of this blog (Part 2).

STRUCTURING COMMODITY PRICE FORECAST PROBLEM

Creating a Predictor Map — Data Collection

Fundamental analysis is the process of collecting supply and demand data and is an essential exercise for commodity price forecasting. It helps in understanding whether a market is in deficit, equilibrium, or oversupply. In addition to supply and demand, there exist other exogenous factors also which can change supply and demand characteristics. For example, political events, sudden imposition of a tariff on imports and exports by a government, sudden changes to currency etc. The major challenge in commodity price forecasting is to find the right and updated data sources describing the market and its participants, and then to understand how these factors dynamically change the models. Some of the examples of supply side, demand side and exogenous factors along with the source’s information are listed below with an example of predictor map shown in Fig.2.

Fig. 2. Example of a predictor map

1) The Supply Side: In general, the supply side of a commodity forecasting problems needs to consider the following indicators:

• Price of raw materials used to manufacture the commodity

• Import, export data from different countries dealing with said commodity

• Level of stockpiles or inventories held in storage of the commodity

• Global operating capacity for the plants manufacturing the commodity

2) The Demand Side: Commodity demand is ubiquitous as the consumption of raw materials occurs all over the world and is highly correlated and impacted by some of the economic factors like currency values, interest rates and economic growth. For ex: commodity demand tends to increase when the world economy is healthy and growing, commodity demand tends to increase, and the converse is true during times of economic weakness. Examples of demand side predictors are listed below:

• Product data (price/stock) for products that chiefly consume the commodity for value addition

• Stock prices of the businesses using the commodity

• Prices of alternative product or competitor of the product manufactured using the commodity

3) Exogenous factors: Some of the examples of exogenous factors are listed below:

• Indices capturing some of the important political event, tariff imposition, change in policy
• Trade war data by capturing sentiments through Twitter or any other source
• Weather data (hurricane) to capture the effect of change in weather for commodity
• Logistic costs
• Currency movements
• Index related (Baltic Dry Index, S&P 500)
• Macroeconomic (Industrial Production)
• Satellite imagery (e.g. measures of activity derived from images of open cast mines)

Fig. 3. Example of few exogeneous factors

These exogenous factors together with fundamental supply and demand data result in the price solution for commodity prices. For most of the important commodities tremendous amount of data is available from both government and private agencies. Ex. American Petroleum Institute (API) which is a private enterprise and the Energy Information Administration (EIA), a US government agency for energy commodities like crude oil, oil products, coal, natural gas, and other energies. Other sources are Argus Media, Bloomberg etc.