Bootstrapping
Following variable sampling plan standard like ISO 2859–1, of which its section 7.2 Differences (b) there is a normal distribution requirement of measured characteristic. However that might not be applicable in some occasions in which either limited samples available or skewed distribution is. Bootstrapping or “Resampling” provides an approach to such estimation.
What is “Bootstrapping”?
For readers can read Chinese, I would recommend an introduction from this blog: http://yenchic-blog.logdown.com/posts/257912-a-brief-talk-on-bootstrap.
In short, the bootstrap method is a resampling technique used to estimate statistics on a population by sampling a dataset with replacement. It is mainly used for estimating standard errors and bias, obtaining confidence intervals, and sometimes for tests.
Ideal “Estimator” would converge to the interested parameter of population as numbers of samples increase (we call this property as “consistency”). An example is sample mean like stated in CLT. However, we don’t know the exact standard errors are as well as the variance of the estimator from this random sampling result even with its consistency property since there is only one dataset and we can’t get the variance of estimator from one estimated value. But we can keep replicating same size datasets by drawing samples with replacement from original sampling dataset. Each new dataset can come out a new estimator value and with these bootstrap estimators we can count the variance of original estimator.
It sounds straightforward but the theoretical proof is beyond the knowledge here. And there is another important property which is regardless of the distribution of population statistics like the CLT.
Deep dive of bootstraping
There is another good article “What Teachers should know about Bootstrap” which is worth of reading. There are some educational examples, comprehensive visual diagrams and analysis for the different scenarios of bootstrap application. A very important and solid concept learned from this lesson is “Inference, Not Better Estimates”. It echos with previous section that in fact, we will not get new or better estimate from the only sampled data by this skill and the only additional information is the distribution of the estimator.
The other learning from this article is the effect and applicability from sample sizes and different distributions. The bootstrap confidence interval width varies sample sizes substantially. The larger the sample size is, the narrower the confidence interval width is. So in the Verizon example mentioned in that article, the CLEC sb =3.97 is smaller than sigma/sqrt(n)=19.5/sqrt(23)=4.07, sb means standard error of bootstrap. And in it’s Fig. 9 example by an exponential distribution which sampling paramter Xbar is highly propotional to it SE (Mean-Variance Relationship), you could find the shape and location of its bootstrap pencentile varies significantly depending on the observed sampling distribution but very similar of its bootstrap t-distribution.
It also provides the formula to count the number of bootstrap samples, for example like within 10% variation of 95% confidence interval. In such case, it would take around 15,000 to estimate cdf of bootstrap distribution to fall within 2.25% and 2.75% with 95% probability.
Bootstrap by SAS
We can implement such resampling skill by SAS and your can refer to this SAS blog to do it step by step. The data is available in Sashelp Iris dataset so we can simply execute the code in SAS OnDemand for Academic.
The summary of this example is as below:
- The bootstrap estimate of the skewness is 1.18. This value is computed as the mean of the bootstrap distribution.
- The bootstrap estimate of the standard error of the skewness is 0.38. This value is computed as the standard deviation of the bootstrap distribution.
- The bootstrap percentile 95% confidence interval is computed as the central 95% of the bootstrap estimates, which is the interval [0.49, 1.96].
And you can see that bootstrapped statistics “skewness” in this case is just a little bit positive skewed but it should be a coincidence and mostly should be normally distributed as CLT.
The application of bootstrap on ML-Calculate Classificatin Accuracy Confidence Interval
We can apply this skill to estimate the confidence interval of classification accuracy. For example, considering we have a trained model and a set of splitted test data and we can get scores and lays as a distribution between 0 and 1. Since we have only one test set then, how would we know in real world like for binary label “True”? how accurate it would be and if we can say like a range of 95% confidence interval? So with this skill, we can resample with score by labels and count the count the mean of sampling distribution and standard deviation then the confidence interval like previous section. For label “True”, it’s a skewed distribution so know some limitation might be there when making estimation. You can refer to this blog for exact Python Decision Tree classification model example.
The mission of GLInB is to bring most value by virtualization of your supply chain quality function to fit for challenges in today’s business environment.
Please visit us at http://www.glinb.com for more information.
