Iterated Reweighted Least Squares and GLMs Explained

With a detailed implementation in Python

Image By Tine Ivanič on Unsplash

Generalised Linear Models (GLM) are regression models where we generalise the linear assumption of the usual linear regression model. Because of this non-linearity, estimating the regression parameter will not be as simple as estimating a linear regression parameter.

The Iterated Reweighted Least Squares (IRLS) algorithm or sometimes also Iterated Weighted Least Squares (IWLS), is a method to find the maximum likelihood estimates of generalised linear models. It is an extension of the weighted least squares method. Let’s start with a short background introduction.

Background

In a linear model, we can estimate the parameter of the regression using the normal equations,

This method will have errors with means of zero and constant variance.

If the relationship between the predictors and the predicted is not linear, we will obtain errors with an unconstant variance if we insist on using the above normal equation.

This result is not desirable. We want to have constant error variance for better predictability. With some mathematical manipulation that I will not show, we can turn the error distribution to a standard multivariate normal…