A simple checklist can (nearly) always uncover bad analytics.

  1. Are there preexisting differences between groups? (i.e., Could groups be probabilistically not equivalent?)
  2. Is there a common driver of both managerial decisions and outcomes? (i.e., Are decisions and outcomes both responding to a common catalyst?)
  3. Is there reverse causality? (i.e., Did managerial decisions change outcomes or did outcomes change managerial decisions?)
  4. Is there a confound that could explain the outcome? (i.e., What else might be going on?)

Remember, interpretation is as important as analysis.