Bias-Variance Trade-off(Overfitting/ Underfitting) and Polynomial Regression
Bias-Variance Trade-Off
During machine learning lessons, the ideas of underfitting and overfitting, together with the bias-variance trade-off, frequently occur. We can better comprehend the machine learning and testing process and contribute to the development of better models by having a thorough understanding of these ideas. In order to make predictions using the features we have, as you might recall from the last part, we construct functional structures or curves. We receive hints about the effectiveness of our model from these functional structures.
Our model needs to learn the fundamental structure of the data in order to be effective and balanced. It is essential for our model to understand the fundamental structure of the data. It will help us estimate the new data easily and accurately using our model. The image below shows three different types of functional structures (models).
Underfitting and Overfitting Problems:
1- High Bias and Underfitting Problem:
The distance between real values and estimated values essentially expresses a bias. It is clear from the second graph below that there are significant gaps between a lot of real data and our curve. Additionally, our curve’s (dotted line) normal distribution cannot be reconciled with the data. There is significant favouritism for certain observations.
This indicates that the curve we produced using our model does not properly represent the data. We can state that our model has not learned completely in this case. This condition is called underfitting.
2- High Variance and Overfitting Problem:
Variance is a measurement of a model’s flexibility and precision. As you can see in the third graph above, with high variance models, they mapped the dataset perfectly rather than learning its structure. Although initially, this would appear to be a beneficial thing, it really leads to the model over-learning or memorization.
As we previously discussed, a successful model is to learn the general structure of the data, not memorize it. Overfitting will provide a model that is specific to those data, this model is not a general one. When the model encounters new data, it will generate incorrect results because it is not designed for the new data.
Example: Underfitting and Overfitting Students
Consider two students who take an exam. The first student just makes cursory preparation for the test and lacks subject mastery. This student will not be able to respond to the questions in a way that is both clear and accurate. We can conclude that this student could not learn the subject exactly, which is underfitting.
Suppose that the second student memorizes all the previous exam questions rather than comprehending the subject. This student will also make a mistake in the first question one encounters because one does not learn the general structure of the subject and only take the exam by memorizing the questions, which is overfitting.
Training Error vs. Validation Error (Test Error)
In the Linear Regression, we explored train and test (validation) errors. Let’s now quickly analyze how these errors relate to overfitting and underfitting as well as how they can be used to create a balanced and effective model.
The ability of our model to provide accurate predictions with lower error rates is one of the key success factors. We may alter many of the parameters and features of our model, which we will employ in the future, to lower the error rates. While these adjustments will reduce the error rate and help our model improve; on the other hand, it makes our model more complex.
While trying to reduce the error rates, we also make our model more complex and put it at the risk of high variance, that is, overfitting. This paradox is exactly the meaning of the bias-variance trade-off.
This paradox has a balance area between the high bias and high variance, as it is seen from the image above. The model’s success is not that the training error is too low, but this is where the training and test errors are in balance. The point that a successful model should have in this balance area.
Polynomial Regression
Polynomial Regression gives the best estimation of the connection between the dependent and independent variables.
The polynomial models can be used when the relationship between the dependent and independent variables is curvilinear. In some cases, polynomials can also be used to show a nonlinear relationship with a limited set of dependant variables.
Simple artificial estimators built on existing data are polynomial regression models. Our fit line behaves nonlinearly as a result of the mathematical tricks we employ to produce new features.
The degree of the polynomial must be considered in polynomial regression models. It significantly influences how complex the model is and, if the degree is high, might result in overfitting. On the other hand, if degree = 1, it is equivalent to simple linear regression.
