Model Accuracy
Model accuracy is typically assessed by measuring the quality of fit (mean squared error, typically).
We can increase fit of a model by increasing the degrees of freedom (higher DFs are wiggly curves, lower DFs are smoother; linear regression has 2 DF).
There’s a tradeoff though — overfitting to training data can result in higher MSE for test data, because training data might not be representative of f.
The theory behind this is called the bias-variance trade-off.
Variance of a statistical model refers to “the amount by which f’ would change if we estimated it using a different training data set” (pg. 34, ISL). Generally, the more flexible a method is, the higher the variance.
Bias of a statistical model refers to “the error that is introduced by approximating a real-life problem” (pg. 35, ISL). For instance, linear regression assumes a linear relationship, which may not be true. Therefore, more flexible models (which don’t assume function form) will have less bias.
Navigating this tradeoff is an art — we’ll probably touch on this later on (with methods like cross-validation).
Classification Accuracy
So how does model accuracy work in classification situations, where output is categorical? we look at training error rate (fraction of incorrect classifications). Specifically, we want to look at the test error rate.
Now the ideal algorithm to use for classification is the Bayes Classifier. This algorithm assumes that we know the conditional distribution of Y given X. For instance, you have a group of Asians (Korean and Chinese), and are trying to categorize them based on nationality. We know that there are more Chinese women than Chinese men, and more Korean men than Korean women. If you know those distributions, then we would also know the conditional probability of P( Nationality = Chinese | gender = woman). If we had this prior information about other attributes as well, then we’d easily arrive at an excellently low error rate, the Bayes error rate.
But the reality is that real world does not give us conditional distributions, so it’s not possible to use the Bayes Classifier. But since the Bayes Classifier is the irreducible error rate, so we’ll strive to get as close to that as possible. Now what are some other ways?
K-Nearest Neighbors (KNN)
Imagine you’re traveling across a foreign land, and trying to figure out where the boundaries between countries are. You travel to a coordinate, and pick the nearest 3 towns and analyze the language and culture. If 2 of the towns speak X and the other town speaks Y, then there is 2/3 likelihood that the point you’re standing at is still in country X. It kind of looks like this:
