Regression, Classification and Confusion Matrix
In the second article, we discussed the types of variables in machine learning, starting with reinforcement learning, supervised learning, unsupervised learning and problem types.
In this article, I will discuss methods for assessing model success.
Error and Performance Evaluation Methods for Regression Models:
- Mean Absolute Error (MAE): Calculates the average absolute difference between the actual values and predicted values. A lower MAE value indicates better performance of the model.
- Mean Squared Error (MSE): Computes the average of the squared differences between the actual values and predicted values. MSE can have higher values compared to MAE. A lower MSE value indicates better performance of the model.
- Root Mean Squared Error (RMSE): Calculated by taking the square root of MSE. It measures the average magnitude of the difference between the actual values and predicted values. A lower RMSE value indicates better performance of the model.
- R-Squared (R2): A statistical metric that indicates the proportion of the variance in the dependent variable that is predictable from the independent variables. An R2 value close to 1 suggests a good fit of the model to the data.
Error and Performance Evaluation Methods for Classification Models:
- Accuracy
- Precision
- Recall (Sensitivity)
- F1 Score
- AUC-ROC Curve
Let’s examine the Confusion Matrix before these values.
What is the Confusion Matrix?
A confusion matrix, also known as an error matrix, is a tool used to evaluate the performance of a machine learning or statistical classification problem.
A confusion matrix is a square matrix that illustrates the relationship between the true and predicted classes in a classification model.
- True Positive (TP): The number of instances that are actually positive and are correctly predicted as positive by the model.
➜ Let’s say the model correctly predicts 80 patients as having the disease, and indeed, they are actually positive for the disease. This would be the True Positive (TP) value in the confusion matrix.
- False Positive (FP): The number of instances that are actually negative but are incorrectly predicted as positive by the model.
➜ Now, let’s say the model incorrectly predicts 15 patients as having the disease when they are actually negative for the disease. These are false alarms or false positives. The False Positive (FP) value in the confusion matrix would be 15.
- True Negative (TN): The number of instances that are actually negative and are correctly predicted as negative by the model.
➜ Let’s say the model correctly predicts 300 patients as not having the disease, and indeed, they are negative for the disease. This would be the True Negative (TN) value in the confusion matrix.
- False Negative (FN): The number of instances that are actually positive but are incorrectly predicted as negative by the model.
➜ Lastly, let’s say the model incorrectly predicts 5 patients as not having the disease when they are actually positive for the disease. These are cases where the disease is missed. The False Negative (FN) value in the confusion matrix would be 5.
Actual: Disease | TP = 80 | FN = 5
Actual: No Disease | FP = 15 | TN = 300
Type I Error: A Type I error, also known as a false positive, occurs when the null hypothesis is true, but we mistakenly reject it in favor of the alternative hypothesis.
➜ The model incorrectly predicts patients as having the disease when they are disease-free.
Type II Error: A Type II error, also known as a false negative, occurs when the null hypothesis is false, but we fail to reject it and mistakenly accept it.
➜ The model incorrectly predicts patients as not having the disease when they actually have the disease.
Accuracy: The basic metric for measuring classification accuracy. It calculates the ratio of correctly classified samples to the total number of samples. However, accuracy may not be sufficient in cases of imbalanced classes or different costs of errors.
Precision: Indicates the ratio of true positive predictions to the total positive predictions. It is important to minimize false positive predictions.
Recall (Sensitivity): Measures the ratio of true positive predictions to the total actual positive instances. It is important to minimize false negative predictions.
F1 Score: The harmonic mean of precision and recall. It provides a balanced evaluation metric, particularly in situations with imbalanced classes or different costs of errors.
AUC-ROC: The AUC-ROC (Area Under Curve) value represents the area under the receiver operating characteristic curve. It is used to assess the accuracy of the classification model and may be preferred over accuracy in cases of class imbalance.
I will end here, in the next post I will discuss overfitting, underfitting and model complexity.
Hope to be in the moment and follow along. 🌞
You can connect with me through my Linkedin network. 🤸🏻
👇🏼👇🏼👇🏼
