Setting a foot in the garden of forking paths

I really like the expression a garden of forking paths. It’s a compelling visual of how you can analyse a dataset in many different ways. This plurality has serious consequences for claims of statistically significant effects.

If I’m not mistaken, it is Andrew Gelman who first referenced Borges’ short story with the same title. Here is the original paper. The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time.

Gelman and Loken write:

Our key point here is that it is possible to have multiple potential comparisons, in the sense of a data analysis whose details are highly contingent on data, without the researcher performing any conscious procedure of fishing or examining multiple p-values.

I really like the expression because it is an image that sticks. Exploring a dataset means setting a foot in the garden of forking paths. Remember you may not even see the other paths.

Another thing to keep in mind when exploring.