Predicting the future using Machine Learning part II
The polynomial model + implementation in Python
In this second part of the “Predicting the future using Machine Learning”-series I would like to elaborate further on finding a better generic model for the same dataset I worked with in the previous part.
The polynomial model
To form a polynomial model, polynomial data is needed. This can easily be retrieved by extending the linear dataset I used earlier to a higher order polynomial dataset by expanding the original input with the original input to the power of a higher order polynomial and adding additional weights to the model.
I wrote a function that creates a polynomial dataset as a matrix given the datapoints and the specified order, which I called k. For ease of calculation I expanded the dataset with a vector of 1’s, which represents the constant weight w0:
I chose order 2, so my polynomial dataset looked like this:
The hypothesis for a single input is of the form:
g(x¹) = ∑ᵢᵏ Dᵢ¹ wᵢ
where g(x¹) = generic model, i = iteration and has a beginvalue of 0, k = order of polynomial, D = D_matrix and w = weight
This can be written as a matrix multiplication for all inputs in a single equation:
I wrote 2 functions to compute the polynomial model, called poly_val and poly_model, separating the calculation into 2 steps. The poly_val function calculates the values for all inputs with the matrix multiplication shown above and the poly_model function takes the D_matrix and the weight vector and computes the corresponding vector of hypotheses:
Computing the cost with the cost function of this model is the next step. I called this function poly_cost and it returns the total cost using this equation:
Implementing this in Python gives:
Solving all partial derivative equations per weight can easily be done with matrix operations. The principle is exactly the same as for the linear model, where we also just compute w0 and w1. So the final equation for the weight vector is of the form:
With the numpy built-in functions for the transpose and inverse, this equation can easily be implemented in Python, like so:
Lastly, I wrote a function that plots the polynomials in a graph, called poly_plot. I made a function out of this, because 2 points are not sufficient to plot the fit, for it is not linear. The easiest way to solve this is to generate a lot of x-values as samples and compute the corresponding y-values and plot them. The function linspace can be used for generating the samples and the more samples, the smoother the line. Applying the poly_val function on these samples will provide the corresponding y-values:
The only thing left now is visualizing the fit:
As can be seen in the graph above, the polynomial fit is a way better fit to this dataset than the linear fit. The disadvantage that occurs with the polynomial model is that it is prone to overfit. In the next part of this series I will discuss a method that can be applied that chooses a model which is less prone to overfit than the polynomial model.
