Polynomial Regression with Keras

Polynomial regression is the basis of machine learning and neural networks for predictive modelling as well as classification problems.

Regression

Regression is all about finding the trend in data (relationship between variables). This allows us to better understand the data distribution and to predict the values for new input variables.

Linear Regression

Linear regression is used when the trend in the data is linear, i.e., it is of the form y = ax + b The other, more commonly used form of regression is polynomial regression.

Polynomial Regression

Polynomial regression used when the trend in data is a polynomial of degree ’n’, i.e., it is of the form y = ax^n + bx^n-1+ … + n.

Now, let’s jump into the code for modelling polynomial regression using Keras. The first block shows all the libraries required for this tutorial.

import numpy as np
import matplotlib.pyplot as plt
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from sklearn.preprocessing import PolynomialFeatures

Lets generate our data. As you ca see, we are using a cubic equation here. We also rescale the data in the range 0–1. Otherwise, the y values are in the thousands, causing the training to take too long. Even after 500 epochs, the loss was in the magnitude of thousands.

def generate_data():
    X = np.arange(-30, 30, 1)
    y = 9*X**3 + 5*X**2 + np.random.randn(60)*1000
    return X, y
trX, trY = generate_data()
trX = trX/max(trX)
trY = trY/max(trY)
#plot the data
fig, ax = plt.subplots()
ax.scatter(trX, trY)

The generated data

The model expects an input with as many features as the degree of the curve. So, we must generate more features from the data. Scikit-learn has such a method.

n = 3
trX_expanded = np.expand_dims(trX, axis=1)
poly = PolynomialFeatures(n)
# returns: [1, x, x^2, x^3]
trX_expanded = poly.fit_transform(trX_expanded)

First, let us try to use a linear model for this. The model is of the form wx + b.

inp = Input((1)) 
out = Dense(1)(inp)
model_linear = Model(inputs=inp, outputs=out)
model_linear.compile(
optimizer=Adam(lr=1e-3), 
loss="mean_squared_error")

Let’s train the model, and plot its curve.

model_linear.fit(trX, trY, epochs=500)
ax.scatter(trX, trY)
ax.plot(trX, model_linear.predict(trX), color="red")

Linear model trying to fit curve to data

Now let’s try polynomial regression.

inp = Input((n+1)) 
#since one of the features is 1, we need an extra input
out = Dense(1)(inp)
model = Model(inputs=inp, outputs=out)
model.compile(optimizer=Adam(lr=1e-3), loss="mean_squared_error")

Training the model, and plotting its final prediction for the dataset:

model.fit(trX_expanded, trY, epochs=500)
ax.scatter(trX, trY)
ax.plot(trX, model.predict(trX_expanded), color="red")

Polynomial regression curve

As we can see, in this case polynomial regression is much better, as the data trend is not linear. Other such cases can be predicting the price of a house based on certain factors, etc. However, in certain situations linear regression is better suited. For example, predicting how much fuel a drive will take, or finding the relation between the age of an appliance and its price.

One thing to keep an eye out for is overfitting, which occurs when the model fits the curve too perfectly. This may sound like a good thing, but what actually happens is that the model gets too accustomed to the training data. When we present it with new data, the training data-specific curve won’t work too well.

Real World Example

Here is a fictional dataset of company positions and their salaries. Let us read and plot the data.

import pandas as pd
filepath = "path/to/dataset.csv"
data_orig = pd.read_csv(filepath)
x_data = data_orig["level"].values
y_data = data_orig["salary"].values
x_data = x_data/max(x_data)
y_data = y_data/max(y_data)
x_data_expanded = poly.fit_transform(np.expand_dims(x_data))
fig2, ax2 = plt.subplots()
ax2.scatter(x_data, y_data) #plot the data

Given Dataset

Now, let’s plot what our model predicts, against the data:

ax2.plot(x_data, model.predict(x_data_expanded), color="red")

The curve can easily be improved if we retrain the model with this data.

This wraps up the tutorial on polynomial regression. It teaches the core concepts behind machine learning, and is a good starting point in the field!