Support Vector Machines Explained

Support vector machines (SVMs) are a popular linear classifier, the current version of which was developed by Vladimir Vapnik and Corinna Cortes. SVMs are supervised learning models, meaning sample data must be labeled, that can be applied to almost any type of data.

They are especially effective at classification, numeral prediction, and pattern recognition tasks. SVMs find a line (or hyperplane in dimensions greater than 2) in between different classes of data such that the distance on either side of that line or hyperplane to the next-closest data points is maximized.

In other words, support vector machines calculate a maximum-margin boundary that leads to a homogeneous partition of all data points. This classifies an SVM as a maximum margin classifier.

On the edge of either side of a margin lies sample data labeled as support vectors, with at least 1 support vector for each class of data. These support vectors represent the bounds of the margin, and can be used to construct the hyperplane bisecting that margin.

Equations 2 and 1 represent the formulas for a line or hyperplane respectively. For all sample data x, an SVM should find weights such that the data points will be separated according to a decision rule. To elaborate, lets assume we have a set of negative and positive values in a two-dimensional euclidean space, along with an initial straight line (drawn in green) between the two classes of data points:

The yellow space in Fig. 1 shows the margin between the points of opposing classes that are closest to each other. The sample points that are circled are the support vectors.

Now, we imagine a vector w, of arbitrary length constrained to be perpendicular to the median of that margin. Then, we take an unknown vector u, and we want to determine whether it is on the + or the - side of the margin. To do this, we project u onto our perpendicular vector w, which gives us the distance to w.