Statistics 101: The MAGIC criteria

About 100 years ago, Ronald Fisher introduced statistical hypothesis testing. Fisher knew what he was doing and, in the situations he was involved in (testing fertilizers and such), what he was doing made sense. But those methods got applied much too widely and people quickly started complaining.

More recently, the complaining has started to take effect, with important groups noting the problems with significance testing and p values. But … if not that, what?

The MAGIC criteria are put forth in Statistics as Principled Argument by Robert Abelson. It’s an easy read, with few formulas but lots of wisdom. I urge those interested in this stuff to go buy a copy.

Abelson lists five criteria by which to judge a statistical argument. He calls them the MAGIC criteria
1. Magnitude How big is the effect?
2. Articulation How precisely stated is it?
3. Generality How widely does it apply?
4. Interesting How interesting is it?
5. Credibility How believable is it?

We can tell how big an effect is through various measures of effect size. We will get into some of these in later diaries, but some of the common ones are correlation coefficients, the difference between two means, and regression…