Quantile Regression and Prediction Intervals
Prediction Intervals
Normally with regression, we minimize the RMSE error to get a point estimate of some interdependent variables. However, we can hardly measure certainty of the value. Let us see an example.
As we can see from the image above, a model can learn parameters differently and hence give different future values i.e. there is some variation within the forecasts. In most forecasting scenarios, the variations will increase with the length of the forecast period. This means the further ahead we forecast, the more uncertain we are.
To quantize the uncertainty of the forecasts, we use prediction intervals. So, instead of just a single value, we now also have a interval like in Fig 2 where we can be confident to a certain degree that the original value lies inside it. Prediction intervals are typically a function of how much data we have, how much variation is in this data, how far out we are forecasting, and which forecasting approach is used. We can see from the image below that if we want to be more confident, then we will have a larger interval. This is there is a trade-off in the values and ranges.
For h-step forecast and output variable y with estimate of the standard deviation σₕ, the prediction interval can be calculated as :
The constant c depends on the coverage probabilities. The values are tabulated here.
Quantile Regression
Another way of generating prediction interval is through quantile regression. As opposed to linear regression where we estimate the conditional mean of the response variable given certain values of the predictor variables, quantile regression aims at estimating conditional quantiles (typically, median) of the response variable.
Least square regression is based on an assumption that residuals (y - y’) have constant variance across values of independent variables. We can not trust linear regression models and its prediction interval which violate this assumption. Quantile Regressions come into rescue since regression based on quantile loss provides sensible prediction intervals even for residuals with non-constant variance or non-normal distribution.
Just as regressions minimize the squared-error loss function to predict a single point estimate, quantile regressions minimize the quantile loss in predicting a certain quantile. Quantile loss is defined as:
where, γ is the required quantile and has value in range (0, 1).
Now, if we train the model for the median i.e. 50th quantile, the quantile loss is simply the sum of absolute errors. To create a prediction interval we can now use other other quantile values. For example in the image below we have 0.9 77and 0.023 percentiles. This gives a prediction interval with 0.95 probability of having the true value within its bounds.
Conclusion
The output value of regression models are subject to uncertainty which can be modeled by prediction intervals. Also, linear regression models cannot be used for datasets with residuals that have non-constant variance or non-normal distribution. This problem is addressed by Quantile Regression.
References:
- Forecasting: Principles and Practices
- Forecasting at Uber: An Introduction
- https://heartbeat.fritz.ai/5-regression-loss-functions-all-machine-learners-should-know-4fb140e9d4b0
- https://www.wikiwand.com/en/Quantile_regression#/Conditional_quantile_and_quantile_regression
- https://www.evergreeninnovations.co/blog-quantile-loss-function-for-machine-learning/
