A simple checklist can (nearly) always uncover bad analytics.
- Are there preexisting differences between groups? (i.e., Could groups be probabilistically not equivalent?)
- Is there a common driver of both managerial decisions and outcomes? (i.e., Are decisions and outcomes both responding to a common catalyst?)
- Is there reverse causality? (i.e., Did managerial decisions change outcomes or did outcomes change managerial decisions?)
- Is there a confound that could explain the outcome? (i.e., What else might be going on?)
Remember, interpretation is as important as analysis.