There Are No Straight Lines In Nature
Which means using linear machine learning models probably won’t model the real world very well.
This article will take a look at a non-linear data set, concentric circles, and see how we can apply a couple of machine learning and deep learning models to create a classification model.
There are many ways to approach this problem, and this article is not meant to be a complete body of work on the subject, but instead a more casual look at how to approach the problem.
DataSet
The dataset was created with Scikit-Learns, make_circles, function. This creates a 2-class classification data which can clearly not be separated by a line. In this case we will need to add some non-linearity to our models.
First — lets see what happens when we try to use a linear model to separate this data.
LogisticRegression
As you can see — LogisticRegression is only able to separate the data as best it can along a straight line. This model wont do.
LinearSVM
Lets look at a linear Support Vector Machine. Because this is also a linear model we would not expect it to perform any better. However we will also see that we can make Support Vector Machines perform very well.
As expected this too does not perform well.
Support Vector Machines have a ‘kernel’ parameter which allows for the model to perform higher dimensional analysis on the data. By adding higher dimensional features to the existing data allows for the separation. There are two kernels we will look at: poly and rbf.
Support Vector Machine with Poly Kernel
Using a Support Vector Classifier with a poly kernel and a degree parameter of ‘3’ we can see that the model is able to separate the data very well and has a holdout accuracy of about 98%.
Support Vector Machine with Radial Bias Function Kernel
Using a Support Vector Classifier with a ‘rbf’ kernel also performs very well with a holdout accuracy of about 98%.
RandomForestClassifier
RandomForestClassifiers is another model that can handle non-linear data. As we can see in this dataset, while it does pretty well it struggles to literally fit a square peg into a round hole. The holdout accuracy for this model is about 90%. Adjusting hyperparameters can result in a higher accuracy but you have to be careful with overfitting.
Deep Learning Neural Network
By their very definition neural networks handle non-linear data. Using a simple Keras like the following:
Training for 600 epochs shows a loss/accuracy graph as below:
With a result like:
As we would expect, a neural network was able to handle the non-linearity in the dataset with enough epochs.
Summary
To see the notebook that was used for this article see my Github Repo.
This article used a non-linear dataset of concentric circles and looked at a few different models that could be used to separate the data. For small to medium sized datasets, Support Vector Machines can be an excellent choice. For larger datasets — neural networks would work well with enough training time.
