Bayesian Style: BEST vs. t-test in [R]

This week’s paper argues that many researchers choose Bayesian models to describe how people perform cognitive tasks, however they deploy non-Bayesian reasoning methods to make inferences from data. Based on my knowledge, I know that by “mortmain of 20th century”, and “traditional norms of behavior”, the author is trying to refer to frequentism. Then, the author mentions some problems of non-Bayesian methods such as vague so-called p-value, poor estimates of parameter values, and too many inappropriate computational constrains and assumptions. As a solution to these deficiencies, the author promotes Bayesian analysis with many advantages: no p-value, rich information for parameters, and customizable models for different data. He continues that Bayesian models for mind might lose their favor some day but Bayesian data analysis will remain appropriate.

Despite my previous knowledge about Bayes’ theorem and conditional probability, this week’s reading sounded new to me. Therefore, I decided to google it in order to grab a big picture. Here we are: It is named after Thomas Bayes (1701–1761), a theory in which the evidence about true state of the world is demonstrated by using “degrees of belief” or Bayesian probabilities. As Figure 1 suggests, prior knowledge plays a tremendous role in Bayesian data analysis. Soon, I figured out that it is the opposite of Frequentist probability or frequentism. Then, I came across to this statement from this webpage: “ 19th century statistics was Bayesian while the 20th century was Frequentist…”.

Figure 1: Bayesian vs. Frequentism (captured from here)

I understood that we have always three types of data in Bayesian data analysis: Prior, likelihood, and posterior. The cool thing is: “the posterior distribution from a previous study can often serve as prior distribution for subsequent studies” (here). The difference between Bayesian and Frequentism is well addressed by Data Science Insider weblog, as Figure 2 depicts:

D: The Data we’ve already seen
H: The Null Hypothesis
Frequentism: p(D/H) = Probability of having D given the H.
Bayesianism: p(H/D)= Probability of having H, given the data.

Figure 2: Bayesian approach vs. Frequentism

About the author of this week’s article

John K. Kruschke is professor of Psychological and Brain Science, and Adjunct Professor of Statistics at Indiana University. He won Troland Research Award in 2002 for deep insights and empirical evaluations concerning concept formation in connectionist frameworks. His current research interests is about moral psychology.

He has a book (Figure 3), Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan, a blog, and a YouTube channel. For example, in one of his videos and posts, he talks about Bayesian Data Analysis and how Bayesian Estimation can supersede the t-test (BEST). He also has a web app that allows you to run BEST online without installing any extra software.

Figure 3: John K. Kruschke’s Book (captured form Amazon).

Bayesian Estimation in [R]

Knowing that Bayesian Estimation can be used instead of t-test, I decide to run a Bayesian Estimation and a t-test in [R] to see the difference.

1. First, I tried to install its R package,it automatically installed ‘rjags’, ‘coda’, ‘jagsUI’ packages as well.

install.packages(“BEST”)
library(BEST)

2. Then, according to this and this tutorials, I understood that I needed to install JAGS in my computer. So, I installed it from here.

Figure 4: for doing BEST in R we need JAGS (captured from here)

3. The next step is to get data files and codes which are used in his book. You can download them from here and here. Their names are “DBDA2Eprograms.zip” for data files, and “BEST” for codes. Just unzip them and be ready for fun parts.

4. Close your R and RSudio program if you have already opened them. Then find “BESTexample.R” from step 3’s unzipped folder of “BEST”. When your computer asks for a program to open this file, choose RStudio. This will automatically chooses that folder as your working directory, which is necessary to run Bayesian data analysis using BEST.

5. Inside the “BESTexample.R”, scroll down and find y1 and y2, which are our group 1 and group 2 for running BEST (see Figure 5). Change them to your desired dataset.

Figure 5: y1 and y2 are our desired groups to make the comparison.

Here, I found a dataset to compare fuel consumption for US cars versus Japanese cars. Figure 6 shows them. y1 is the group of US cars and y2 is the group of Japanese cars.

Figure 6: miles per gallons (mpg) of US cars (y1) vs. Japanese cars (y2).

Then, I ran a t-test in [R] and then a BEST in [R]. The Code and its result (Figure 7) for t-test are as follows. Based on the result we can conclude that with one gallon, Japanese cars drive significantly more miles.

t.test(y1,y2)

Figure 7: t-test result in [R] using stats package.

Now, let’s run BEST for this dataset: (it took at least 3 minutes). By looking at Figure 8, we can conclude the same result as the former test: Japanese cars are more fuel-efficient. However it has BEST certification this time. We can see that %95 of Highest Density Interval (HDI) (-12.1 to -8.89) reported by BEST is different than %95 of Confidence Interval (-11.91 to -8.75) reported by t-test. Nevertheless, since the mpg difference of US cars and Japanese cars is extensive, both tests suggest the same conclusion.

Figure 8: BEST result in [R] using BEST package.

As a conclusion, I am happy that I learn how to run Bayesian estimates in [R] and I reviewed an example to understand the difference between t-test and BEST. I hope in near future I can use BEST method to take its unexplored advantages.