Regression with Python from Scratch — Polynomial Regression
In this story I generalize my explanations for linear regression into polynomial regression. Again starting by creating random data with noise, this time I use more of a method approach so that we can create a class for regression problems.
Polynomial regression is a type of linear regression, known as a special case of multiple linear regression. It again makes predictions using only one independent variable, but assumes a nth degree polynomial relation between said independent variable and the dependent one.
Our assumption is also valid (just as is for linear regression) that the data that is to be predicted, is the sum of a deterministic function of the input and a random noise.
where r is the dependent variable, x is the independent variable and 𝜖 is the random noise.
Generalizing our estimator used in linear regression, we have
which can also be written as
Let us start with imports.
import numpy as np
import matplotlib.pyplot as pltnp.random.seed(3)
Creating random polynomial data with noise
Generally, our function for the polynomial data can be written similarly to our estimator function.
where 𝑐 are the constants for every term in the function, and k is the degree of our polynomial function.
Remember that we only write this here because we are generating the data that is going to be used in this tutorial. In a real world case, we would not know what type of function our data best fits, which is what we aim to find with our estimator function.
The reason why we use degree + 1 while creating constants is that we also add a bias to our data (c_0).
Let us plot the data.
plt.scatter(X, y, s = 5)
We can methodize data creation part to use easily later.
Generating Training and Validation Sets
Model Training
Random Initialization of Weights
We again start by initializing weights randomly.
Let us plot our not yet trained estimation against our data and see how it does not make a really good fit.
Updating Weights
In order to update our weights, we are going to use the equation
which is the same used for linear regression. Though now, we can generalize these matrices as follows.
Notice that N is again the sum, but with 0 powers of x.
We can implement this matrix dot product as below.
Once again we create our plot to see if this time, our predictions does fit. Let us make two plots, one to see every training data predicted, and the other for validation data.
Error and Coefficient of Determination
We are going to be using the same error that was used in linear regression, which is Relative Squared Error.
Predicting with higher or lower degree polynomial predictors
Now let us take this a step further and see what if we assumed a 2nd degree relation between our data points and their corresponding values. We could calculate our new regressor with weights up to W squared term (remember that python range function is an open one on the upper end, meaning it does not include the upper end).
Which gives us the plot
It is clear that we will have a larger error, since this fitting does a mediocre job predicting our generated values.
Or …
We could do the opposite and predict using a higher degree estimator.
This time though, we expect to have a nice fit to our data, since we expect higher degree weights to be nearly zero.
Let us see if higher degree weights are really close to zero. Printing W variable, we have
as expected.
Model Using Numpy
Numpy is too handy not to use and it has a method for calculating a fit-line. Only using one line of code (after generating our data of course), we can create a fit-line for the data.
As can be seen, we can also make predictions by passing the input data to the model.
References
- Ethem Alpaydin. 2010. Introduction to Machine Learning (2nd. ed.). The MIT Press.
Appendix
Let us put these methods into a class. I have written this class just for this purpose and not to make a library, so please note that this is of course not the best way to write this class; since there is no error handling or optimizing etc.
Now we can just call our methods one by one and see our regression process in action.
Of course we can also repeat the last part where we have used different degree predictor for the data. With our new class, it is easy to do; we only change the degree parameter while creating the random data.
