Leveraging conformal prediction in Python to accelerate the renewable energy transition

Figure 1: On the electricity grid, supply and demand have to be in balance every moment in time

Forecasting the future is inherently uncertain, which can impact our decision-making. This uncertainty becomes increasingly important in the energy markets, where renewable sources like wind and solar are more unpredictable than fossil fuels. The electricity grid must always be in balance, meaning that the electricity supply must exactly match demand at any given moment. The volatility of renewable energy sources makes it challenging to keep the grid in balance. Quantifying this uncertainty improves the decision-making beforehand to prepare for this balance and, with that, accelerate the renewable energy transition.

A common approach is to forecast point estimates, without details about uncertainty. If you decide to use a probabilistic forecasting method, most methods don’t guarantee a calibrated distribution. This can result in incorrectly predicted measures of confidence. Conformal prediction solves these problems. As a data scientist at Dexter Energy, quantifying uncertainty helps me on a daily basis. It can help in any field where a measure of uncertainty is valuable, such as the medical field, operational planning, and more.

This blog explains the advantages of conformal prediction and the computation behind the widely used Python packages. Understanding this computation will help you applying conformal prediction or conformalized quantile regression to your own forecast.

If you prefer listening instead of reading, you can watch my presentation on conformal prediction.

Advantages of conformal prediction

Conformal prediction is an elegant and effective method that can be applied to any forecast. The advantages of conformal prediction are:

1. Valid coverage: Conformal prediction guarantees a valid coverage in finite samples, meaning that the prediction intervals align with the distribution of the data set. This is not the case for most methods that forecast prediction intervals or probabilistic distributions.
2. Model agnostic: Conformal prediction is model agnostic, so it can be combined with any point forecast or probabilistic forecast that you are using.
3. No distributional assumptions: Conformal prediction does not require assumptions about the forecasted distribution beforehand.
4. Easy to implement: The simplicity of conformal prediction makes easy to implement.

Two ways of using conformal prediction

The most common ways of applying conformal prediction are:

1. Prediction interval: Add a prediction interval to a point forecast.
2. Calibration: Calibrate a probabilistic forecast to ensure that on average the predicted distribution matches the actual distribution of the data set.

Figure 2: Adding a prediction interval versus calibrating a probabilistic forecast

Both of these methods can be easily implemented in Python. Common Python packages for conformal prediction are crepes and MAPIE. For an example of how both of these packages implement conformal prediction, see my repository on conformal prediction. The package that is most suitable for your project depends on the goal of the project:

• Crepes package: Best suited for simpler tasks. For example, for adding a prediction interval for a basic prediction with a limited amount of data, the crepes package was 10 times faster. Additionally, the underlying code of the crepes package is easier to read if you want to have a more detailed look at it.
• MAPIE package: Best suited for more extensive tasks. MAPIE is more extensive and offers many possibilities to step into more complex methods, combined with good documentation.

What do these packages compute exactly? In the next sections, I will explain the computation behind the packages, including the corresponding Python code.

1. Add a prediction interval to a point forecast

A prediction interval provides insight into the uncertainty of a point forecast. With conformal prediction, we can add a prediction interval around any forecast.

Figure 3: Adding a prediction interval to a point forecast

How do you compute the prediction interval with conformal prediction? Here are the five steps:

Step 1: Hold out a calibration set from your train set. This calibration set is used to compute the prediction interval.

Figure 4: For conformal prediction, you split your data into one additional data set

Step 2: Train your point forecast model the usual way on the train set.

Step 3: Predict your calibration set. The residuals, the difference between the prediction and the actual value, are your non-conformity scores. To determine the width of the prediction interval, create a sorted list of the absolute values of the non-conformity scores. From this list, select the quantile that matches the confidence level that you want the prediction interval to show. The selected non-conformity score gives the width of the prediction interval.

Figure 5: Computing the width of the prediction interval based on the non-conformity scores

Step 4: Predict for your test set with your trained point forecast model the usual way. When you have the point prediction, add the prediction interval by adding and subtracting the width of the prediction interval that was computed in the previous step.

Figure 6: Adding the prediction interval to your forecast

That’s it! You now added a prediction interval to your forecast. In Python, this computation is:

import numpy as np
from lightgbm import LGBMRegressor
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split

# Preparation: Start with fetching your data set
dataset = fetch_openml(name="house_sales", version=3, parser="auto")
X = dataset.data.values.astype(float)
y = dataset.target.values.astype(float)

# Step 1: Split the data set to train, calibration and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)
X_prop_train, X_cal, y_prop_train, y_cal = train_test_split(
    X_train, y_train, test_size=0.25
)

# Step 2: Train a point prediction model as a baseline
fitted_baseline_model = LGBMRegressor().fit(X_prop_train, y_prop_train)

# Step 3: Compute the width of the prediction interval
confidence = 0.90  # Select the level of confidence you want to apply
prediction_cal = fitted_baseline_model.predict(X_cal)
residuals = y_cal - prediction_cal
conformity_scores = np.sort(np.abs(residuals))[::-1]
quantile_index = int((1 - confidence) * (len(conformity_scores) + 1)) - 1
width = conformity_scores[quantile_index]

# Step 4: Add the prediction interval to your point prediction
point_prediction = fitted_baseline_model.predict(X_test)
prediction_intervals = np.zeros((len(point_prediction), 2))
prediction_intervals[:, 0] = point_prediction - width
prediction_intervals[:, 1] = point_prediction + width

Figure 7: Result of adding a prediction interval in Python

2. Calibrate a probabilistic forecast

Using conformal prediction to calibrate a probabilistic forecast combines the best of two worlds:

1. Statistical guarantee of valid coverage: The results have the guarantee of valid coverage from conformal prediction. Most machine learning models don’t have a guarantee, but are asymptotically consistent. This means that with a feature space with full information about reality the results are correct, but since the feature space is never perfect, it can occur that the results are biased.
2. Adapts to the input space: The predicted distribution adapts to the local variability of this input space, where the prediction interval described in above section was constant over the prediction set.
3. Full distribution: a probabilistic distribution provides more information then a prediction interval.

In the example of quantile regression this is called conformalized quantile regression. How to apply conformalized quantile regression?

Step 1: Hold out a calibration set from your train set. This calibration set is used to compute the prediction interval.

Step 2: Train your quantile regression model the usual way on the train set.

Step 3: Compute the correction for calibration:

• Predict your calibration set.
• Select an interval from the predicted distribution (for example, 20–80% or 1–99%). The calibration will be done iteratively over all intervals.
• For one interval, the residuals, the difference between the prediction and the closest side of the predicted interval, are your non-conformity scores. Create a sorted list of the non-conformity scores.
• From this list, select the quantile that matches the confidence level that you want the interval to show. The selected non-conformity score gives you the correction that you have to apply in the calibration.
• Repeat this for the next interval.

Figure 8: Computing the correction to calibrate your predicted probabilistic distribution

Step 4: Predict your test set with your quantile regression model the usual way. Next, iterate over all predicted intervals and calibrate each interval by adding the correction that was computed in the previous step.

If the correction is positive, that means the forecasted distribution is too narrow. The calibration will result in a wider, so less certain, distribution. If the correction is negative, that means the forecasted distribution is too wide and will be calibrated to a more narrow and more certain distribution.

Figure 9: Calibrating the forecasted distribution

Done! You now made a conformalized quantile regression forecast. In Python the computation is:

import numpy as np
from lightgbm import LGBMRegressor
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split

# Preparation: Start with fetching your data set
dataset = fetch_openml(name="house_sales", version=3, parser="auto")
X = dataset.data.values.astype(float)
y = dataset.target.values.astype(float)

# Step 1: Split the data set to train, calibration and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)
X_prop_train, X_cal, y_prop_train, y_cal = train_test_split(
    X_train, y_train, test_size=0.25
)

# Step 2: Train a probabilistic prediction model as a baseline.
# This example gives quantile regression for single interval,
# for a full distribution you have to iterate over the intervals
lower_bound_model = LGBMRegressor(objective="quantile", alpha=0.05)
lower_bound_model.fit(X_prop_train, y_prop_train)
higher_bound_model = LGBMRegressor(objective="quantile", alpha=0.95)
higher_bound_model.fit(X_prop_train, y_prop_train)

# Step 3: Compute the correction factor for calibration
confidence = 0.90  # Confidence should match the quantile regression interval
quantile_regression_calibration_intervals = np.zeros([len(X_cal), 2])
quantile_regression_calibration_intervals[:, 0] = lower_bound_model.predict(X_cal)
quantile_regression_calibration_intervals[:, 1] = higher_bound_model.predict(X_cal)

conformity_scores = np.max(
    [
        quantile_regression_calibration_intervals[:, 0] - y_cal,
        y_cal - quantile_regression_calibration_intervals[:, 1],
    ],
    axis=0,
)
emperical_quantile = confidence * (1 + (1 / len(y_cal)))
correction_factor = np.quantile(conformity_scores, emperical_quantile, method="higher")

# Step 4: Add the correction factor to your predicted interval
# This example calibrates a single quantile regression interval,
# for a full distribution you have to iterate over the intervals
quantile_regression_prediction_intervals = np.zeros([len(X_test), 2])
quantile_regression_prediction_intervals[:, 0] = lower_bound_model.predict(X_test)
quantile_regression_prediction_intervals[:, 1] = higher_bound_model.predict(X_test)

correction_factor_matrix = np.ones([len(X_test), 2])
correction_factor_matrix[:, 0] = correction_factor_matrix[:, 0] * correction_factor * -1
correction_factor_matrix[:, 1] = correction_factor_matrix[:, 1] * correction_factor

conformalized_prediction_intervals = (
    quantile_regression_prediction_intervals + correction_factor_matrix
)

Table 1: An example of the Python result for a prediction interval of 90% certainty, conformalizing quantile regression improves the coverage from 83% (biased) to 90% (correct)

Note: Conformal prediction for time series assumes exchangeability of the data, which is almost never the case when dealing with time series. There are methods to deal with dynamic time series that apply more weight to recent data. Unfortunately, with these changes, the statistical guarantee of valid coverage becomes asymptotic: the result approaches the correct distribution instead of always having the correct distribution. For more details look into mapie.regression.MapieTimeSeriesRegressor.

Conclusion

We now understand the two most used ways of applying conformal prediction to quantify the uncertainty of our forecast:

1. Prediction interval: Add a prediction interval to a point forecast.
2. Calibration: Calibrate a probabilistic forecast with conformalized quantile regression.

This blog explained the computation behind the widely used Python packages. This will help you applying conformal prediction or conformalized quantile regression to your own forecast. For the implementation with the Python packages, see my repository on conformal prediction.

If you’d like to learn more about the topic, I recommend the book ‘Practical Guide to Applied Conformal Prediction’ by Valeriy Manokhin and the repository awesome-conformal-prediction.