What is a support vector?
Support vector machines are supervised machine learning models that are well-suited for high-dimensional data and still perform well when there are more dimensions than training data. (You do still need to be careful in choosing kernel functions and regularization for those situations, of course.)
So what is a support vector?
Let’s start with the concept of support. In mathematics, the support of a function is the set of elements that don’t map to zero when the function is applied. You may have encountered this before in classification metrics as the number of elements in the data used to calculate the metric that belong to each class — they are the support set for that class.
In SVM, we find the hyperplane that gives a maximal margin between the points of different classes that are closest together. Their proximity to the decision boundary makes them more difficult to classify, and so the hyperplane is defined by these points. This set of points are known as the support vectors.
A key attribute of the hyperplane is its normal vector. This is used at inference to project new data points to determine where they fall relative to the hyperplane. If the dot product of the normal vector and the data point is greater than a constant (the bias of the model), then the data point is assigned the positive class for the hyperplane.
The normal vector of the maximal margin hyperplane is the sum of the support vectors. Thus, if one of the support vectors changes or is removed, the hyperplane and normal vector will change.
Support vectors and other key concepts for linear models in real-world settings are covered in my Machine Learning Flashcards: Linear Models deck. Check it out on Etsy!
