5 Resampling Methods
Keys: Cross-validation & Bootstrap
1. Prediction Error and Validation Set
1. Prediction Error and Validation Set
How do we test methods like regression and classification out?
Usually a new sample, but we don’t always have new data
→ 2 resampling methods: Cross-validation & the Bootstrap
Goal: how well your prediction method works → the test set error of a model
- estimate test-set prediction error
- variance & bias of parameter estimates (Bootstrap)
Validation set approach — Estimate the test error
The Validation Process
Drawbacks of validation set approach
- 2-fold validation is wasteful because we throw half of the data set to the training set
- the validation estimate of the test error can be highly variable
- validation set error may overestimate the test error. Why?
because the more data, the more information, the lower the error
That’s why we need K-fold Cross-validation
2. K-Fold Cross-Validation
Randomly divide data to equal-sized parts, the only difference is by one observation
Leave-one-out cross-validation (LOOCV) k=5 or 10 is better for better bias-variance trade-off
√ smaller bias since each training set is only (K-1)/K as big as the original training set
3. Cross-Validation Do’s and Don’ts
How do we estimate the test set performance of this classifier?
Cross-validation
but how? Can we apply cross-validation in step 2, forgetting about step 1?
No, why no?
If we do so, we fool cross-validation by leaving out the first filtering step and giving it a very cherry-picked set of predictors in step 2.
The Wrong and Right Way
Wrong❌: Apply cross-validation in step 2
Right☑️：Apply cross-validation to step 1 and 2
4. Bootstrap (有放回式取出）
Bootstrap: pull oneself up by one’s bootstrap — obtain distinct data sets by repeatedly sampling observations from the original data set with replacement
At each stage, every ball has the same probability of being sampled and can be sampled > 1
n= 3 observations
1st — get observation 3 twice, and observation 2 didn’t get sampled at all
…
Sum: Boostrap — we don’t have the population and let’s replace the population
Other uses of Boostrap
- obtain standard errors of an estimate (primary)
- confidence intervals for a population parameter
Can the bootstrap estimate prediction error?
In cross-validation, each of the K validation folds is distinct from the other K-1 folds used for training: there is no overlap. — crucial for its success
— — —Reference — — — — — — — — — — — — — — — — — — — — — — — —
