Confidence Interval Interpretations and Misunderstandings

Confidence Interval Interpretations and Misunderstandings

Confidence intervals (CIs)provide a means to judge point estimates based on a sample from the population.

If that statement excites you, you may well have the makings of a fine statistician.

CIs are a form of internal estimate and specify a range within which a parameter may reside. CIs helps us evaluate the trustworthiness of point estimates.

3 Ways to Interpret Confidence Intervals

Let’s use a 90% confidence interval for our discussion.

Here are three ways to interpret a 90% confidence interval:

1. The confidence interval can be expressed in terms of samples (or repeated samples):

“Were this procedure to be repeated on numerous samples, the fraction of calculated confidence intervals (which would differ for each sample) that encompass the true population parameter would tend toward 90%.”[1]

2. The confidence interval can be expressed in terms of a single sample:

“There is a 90% probability that the calculated confidence interval from some future experiment encompasses the true value of the population parameter.”

Note this is a probability statement about the confidence interval, not the population parameter. This considers the probability associated with a confidence interval from a pre-experiment point of view, in the same context in which arguments for the random allocation of treatments to study items are made. Here the experimenter sets out the way in which they intend to calculate a confidence interval and know, before they do the actual experiment, that the interval they will end up calculating has a particular chance of covering the true but unknown value.[3]

This is very similar to the “repeated sample” interpretation above, except that it avoids relying on considering hypothetical repeats of a sampling procedure that may not be repeatable in any meaningful sense.

3. The explanation of a confidence interval can amount to something like:

“The confidence interval represents values for the population parameter for which the difference between the parameter and the observed estimate is not statistically significant at the 10% level”.[6]

In fact, this relates to one particular way in which a confidence interval may be constructed.

4 Ways to Misunderstand Confidence Intervals

I’m not alone in misinterpreting the meaning of confidence intervals. A few studies have shown the nature of the misunderstanding in published works.[7][9][10]

1. A 95% confidence interval does not mean that for a given realised interval calculated from sample data there is a 95% probability the population parameter lies within the interval.[11]

Once an experiment is done and an interval calculated, this interval either covers the parameter value or it does not; it is no longer a matter of probability. The 95% probability relates to the reliability of the estimation procedure, not to a specific calculated interval.[12] Neyman himself (the original proponent of confidence intervals) made this point in his original paper:[3]

“It will be noticed that in the above description, the probability statements refer to the problems of estimation with which the statistician will be concerned in the future. In fact, I have repeatedly stated that the frequency of correct results will tend to α. Consider now the case when a sample is already drawn and the calculations have given [particular limits]. Can we say that in this particular case the probability of the true value [falling between these limits] is equal to α? The answer is obviously in the negative. The parameter is an unknown constant and no probability statement concerning its value may be made…”

Deborah Mayo expands on this further as follows:[13]

“It must be stressed, however, that having seen the value [of the data], Neyman-Pearson theory never permits one to conclude that the specific confidence interval formed covers the true value of 0 with either (1 − α)100% probability or (1 − α)100% degree of confidence. Seidenfeld’s remark seems rooted in a (not uncommon) desire for Neyman-Pearson confidence intervals to provide something which they cannot legitimately provide; namely, a measure of the degree of probability, belief, or support that an unknown parameter value lies in a specific interval. Following Savage (1962), the probability that a parameter lies in a specific interval may be referred to as a measure of final precision. While a measure of final precision may seem desirable, and while confidence levels are often (wrongly) interpreted as providing such a measure, no such interpretation is warranted. Admittedly, such a misinterpretation is encouraged by the word ‘confidence’.”

2. A 95% confidence interval does not mean that 95% of the sample data lie within the interval.

3. A confidence interval is not a range of plausible values for the sample parameter, though it may be understood as an estimate of plausible values for the population parameter.

4. A particular confidence interval of 95% calculated from an experiment does not mean that there is a 95% probability of a sample parameter from a repeat of the experiment falling within this interval.

References

1. Cox D.R., Hinkley D.V. (1974) Theoretical Statistics, Chapman & Hall, p49, p209
2. Neyman, J. (1937). “Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability”. Philosophical Transactions of the Royal Society A. 236: 333–380. doi:10.1098/rsta.1937.0005.
3. Cox D.R., Hinkley D.V. (1974) Theoretical Statistics, Chapman & Hall, p214, 225, 233
4. Kalinowski, Pawel, Identifying Misconceptions About Confidence Intervals, ICOTS8 (2010) Contributed Paper Referred, https://en.wikipedia.org/wiki/Confidence_interval, accessed 12/26/2016.
5. Hoekstra, R., R. D. Morey, J. N. Rouder, and E-J. Wagenmakers, 2014. Robust misinterpretation of confidence intervals. Psychonomic Bulletin Review, in press. [3]
6. Scientists’ grasp of confidence intervals doesn’t inspire confidence, Science News, July 3, 2014
7. Morey, R.D., Hoekstra, R., Lee, M.D., Rouder, J.N., Wagenmakers, E-J. (2015) The Fallacy of Placing Confidence in Confidence Intervals. in press. [4]
8. “1.3.5.2. Confidence Limits for the Mean”. nist.gov.
9. Mayo, D. G. (1981) “In defence of the Neyman-Pearson theory of confidence intervals”, Philosophy of Science, 48 (2), 269–280. JSTOR 187185

The interpretations and misunderstanding text and with cited references are in whole from Wikipedia page titled Confidence interval https://en.wikipedia.org/wiki/Confidence_interval and slightly reformatted here with number list instead of bullet list.

This article is available under Creative Commons Attribution-ShareAlike License. http://creativecommons.org/licenses/by-sa/3.0/

Originally published at Accendo Reliability.