Frequentist vs Bayesian: the case of the Confidence Interval
There are 2 kinds of people in the (statistics) world: frequentists and bayesians.
Frequentist statistics is the classical branch, this is what we learnt in our elementary statistics course at school. For a frequentist, all the parameters are constant, and in order to find its value, we need to consider not only the data that we possess but also the hypothetical data, so that we don’t miss any cases. In other words, frequentist is focused on repetition, thus their name.
A bayesian, in the other hand, believes that everything, including the parameters, is a random variable, which means that the parameters are represented by a distribution, not only a value. This (hypothetical) distribution is updated continuously with the data. Only in bayesian statistics that we can write P(H|D) (probability of a hypothetical distribution given the data), because for a frequentist, a parameter is a constant, and constant doesn’t have a distribution.
Back to the confidence interval, its definition in frequentist statistics is quite tricky. Many people misinterpret that a 95% frequentist CI means that there is 95% chance that our parameter is in that interval. This, under a frequentist viewpoint, makes no sense! Because, as we have pointed out earlier, for a frequentist, a parameter is a constant, thus he can be sure that a parameter is either contained in that interval or it isn’t, it is either 0 or 1, there is no uncertainty, no variation here! So when we say “95% frequentist CI”, what that refers to is not the probability for the data we have, but it is in fact under hypothetical repetition: if we repeat the same procedure over and over again, in 95% cases the interval will contain the parameter. It doesn’t say anything on the specific problem that we are working on. Bayesian, on the other hand, is more focused on the problem at hand, and its interpretation is usually closer to our intuition: a 95% bayesian CI means that there is 95% chance that the parameter falls in the interval.