Evaluating Model Performance via Cross-Validation
To measure regression model performance, we use RMSE which stands for Root Mean Squared Error.
You can do this easily with Scikit-Learn, but at first, lets train our model:
from sklearn.linear_model import LinearRegression# Let’s train our model at first
reg = LinearRegression()
reg.fit(features, labels)# Then we get some set of predictions
predictions = reg.predict(features)
And now let’s calculate RMSE of our predictions:
from sklearn.metrics import mean_squared_error
mse = mean_squared_error(labels, predictions)
rmse = np.sqrt(mse), where rmse our RMSE value.
However, this approach has one flaw: we use the same set of features and labels to make our predictions, which may lead to model overfitting.
Since we don’t want to touch test set yet, we need to reuse our train data set to effectively validate our model without risking to get a 0.0 value of RMSE which means our model got us and is totally overfitted.
And this is were cross-validation comes to rescue
The main idea is to split our train test into a smaller training set and a validation set. Splitting a set manually may be a bit tedious. Luckily, SKlearn has a nifty cross_val_score function which splits our train set into X distinct sets (defined by parameter cv), then it trains and evaluates our model X times which as a result will give us an array containing X evaluation scores:
from sklearn.model_selection import cross_val_score
scores = cross_val_score(reg, features, labels, scoring=”neg_mean_squared_error”, cv=5)rmse_scores = np.sqrt(-scores)
Here’s what this function did behind the scenes: it split our dataset into five chunks called folds. Then it trained our Regression model 5 times, picking a different fold for evaluation and training on the remaining four folds. Which as a result gave us five trained models and five scores.
Check results:
print(rmse_scores) # To get an array of scores
print(rmse_scores.mean()) # To get a mean
print(rmse_scores.std()) # To get a standard deviationFor more info about what cross_val_score function is capable of checkout SKLearn’s docs here: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html
