For Robust Results, Pair Tradition with Bayesian
I have always felt a certain disdain for traditional significance tests due to the way such models expect data to fit in nice and neat little patterns. Especially when data does not fit due to outlier values, these tests force the researcher to contend with making decisions about conforming data to fit a model. People are strange and adaptable, noise makes muddy data, and traditional tests quickly become inadequate.
There are times when the data is nice and neat, so if the traditional significance test fits, wear it. However, it is important to understand that traditional significance tests result in a p value, and this p value is heavily influenced by experimental design and uncontrollable factors. This means reporting a p value alone may no longer be enough. If you follow my posts you may recall reading a bit about this in Why Reporting Human Data is Changing.
More recently, I have written about regression and ANOVA. Both are types of traditional significance tests which result in a p value, and both have assumptions that must be met in order to accurately be used to describe data. Outside of these tests and others like them are alternative methods for analyzing data. This post covers Bayesian analysis.
Bayesian analysis differs from traditional significance tests. Traditional tests take a frequency approach to probability. The frequency approach says that the probability of some phenomenon occurring is calculated by dividing the number of trials where the phenomenon was observed by the total number of trials observed. The total number of trials is the sample size, and the closer the sample size is to the total population, the closer the probability is to the exact probability. These observations are made in randomly assigned experiments.
Bayesian analysis takes a Bayesian approach to probability. The Bayesian approach says that the probability of some phenomenon occurring is calculated by updating parameters for a previously established probability. By starting with a prior probability, one can update to a posterior probability with new and relevant data. New and relevant data is observed in randomly assigned experiments.
Randomly assigned experiments produce random variables, and random variables have an amount of uncertainty associated with them. In a traditional significance test, this is known as the confidence interval. In Bayesian analysis, this is known as the credible interval. It is important to disclose this because it helps draw similarities between these two methodologies. Yes, Bayesian analysis is different from significance tests, but it is not free from rigid mathematics that help validate its results.
The take home message in today’s post is that researchers have many options available to them when it comes to data analysis. Sometimes the data works well with significance tests, but there are additional tests out there that complement well. A combination of appropriate analyses makes for more robust findings.
