How do We Transform into Machine Learning?

Photo by Alex Knight on Unsplash

What is Machine Learning?

Machine learning (ML), a subset of artificial intelligence (AI), is an area of computational science that focuses on interpreting and analyzing data patterns or structures to enable learning, reasoning, and decision making outside of human interaction.

Figure 2: Machine Learning Scope

Machine learning allows the user to feed an immense amount of data to a computer and have the computer analyze and make data-driven recommendations and decisions based on only the input data. If any mistakes are identified, the algorithm can incorporate that information to improve its future decision-making accuracy.

Overfitting vs Underfitting

To derive a solution, we should first understand the problem. Before we proceed to understand cross-validation, let us first understand the difference between overfitting and underfitting.

Figure 3: Overfitting (left, CV score : 0.9926) vs Underfitting (right, CV score : 0.8291)

Overfit Model: Overfitting occurs when the machine learning algorithm captures the noise of the data. Intuitively, overfitting occurs when the algorithm or the model fits the data too well. The overfitting model results in a good accuracy for training data set but poor results on a new data set.

Figure 4: Overfitting

Underfit Model: Underfitting occurs when the machine learning algorithm cannot capture the underlying trend of the data. Intuitively, underfitting occurs when the algorithm or the model does not fit the data well enough. Underfitting is often a result of an excessively simple model.

Figure 5: Underfitting

Why We Use SVM?

Support Vector Machine (SVM) is a part of supervised learning (machine learning) methods. SVM used a statistical method in which train data can be used to produce a decision-maker system that will be used to predict a new value. Error prediction is calculated based on the overall results of the test data and the results are used to evaluate the quality of the model that has been made. The following are the advantages of SVM:

1. SVM works well when there is clear margin separation between classes,
2. SVM is more effective in high dimensional spaces (in cases where the number of dimensions is greater than the number of samples), and
3. It uses a subset of training points in the decision function (called support vectors), so it is also relatively memory efficient.

Cross-validation in SVM

Cross-validation is a statistical method of evaluating and comparing learning algorithms by dividing data into two segments: one used to train a model and the other one is used to validating the model.

In typical cross-validation, the training and validation sets must cross-over in successive rounds such that each data point has a chance of being validated against.

Figure 6: Cross-validation (Evaluating Estimator Performance)

There are three parameters in Support Vector Machine. The explanation of these parameters are as follow :

• C (regularization parameter): C parameter balances the trade-off between the model complexity and empirical error. To simplify, when C is large, the SVM tends to be overfitting while when C is small, the SVM tends to be underfitting.
• γ (gamma): γ parameter determines the SVM output by the kernel function. To simplify, when γ is large, the SVM tends to be overfitting. On the other hand, when γ is small, the SVM tends to be underfitting
• ε (epsilon): ε-intensive loss function affects the smoothness of the SVM’s response and it affects the number of support vectors, so both the complexity and the generalization capability of the model depend on its value.

The parameters above should be tuned to find the best parameter combination by using grid search techniques.

SVM Application

Here is an example of SVM application for the world gold price forecasting. The first step is training the historical data. We use the gold price history from 1950 to 2018 as the train data. In this step, the historical data is trained using cross-validation. Then this model is applied to predict the gold price in 2019 to 2039.

Figure 7: The World Gold Price Forecasting (CV score : 0.9591)

Conclusion

In this article, we did not cover the math behind the SVM. To learn more about it, you can study Machine Learning: A Probabilistic Perspective by Kevin P. Murphy. It might take some time to understand the derivation. You can also visit the sklearn documentation of SVM to understand more about the implementation of the algorithm in Python. If you have any doubts you can post it in the comment section. Thank you :)