We’re not as good at selecting officer candidates as we think

The Army selects officer candidates through its Army Officer Selection Board (AOSB – LINK). The AOSB is essentially a prediction by army staff about whether a particular young man or woman will become a good army officer.

Does this system work? Does it help the Army select officers who are able to bear up in the combat? To be rigorous, there would need to be two questions: “What percentage of officer candidates who pass AOSB go on to be good army leaders? What percentages of officer candidates who fail AOSB would be successful as army leaders if they had been passed?” These questions would test whether AOSB works, but it would also be interesting to see (for fun) whether the system is fundamentally robust by testing whether candidates who pass AOSB would pass again, if re-tested.

I would be interested to see data on the subject – assuming the Army actually keeps track.

Daniel Kahneman doesn’t think this system is very predictive. Kahneman (LINK) is a Nobel prize-winner and one of the leading psychologists in the world. He got his start in the Israeli Army, where he evaluated officer candidates on the Israeli equivalent of the AOSB. The two are very similar, with the usual battery of tests and leaderless tasks. Kahneman, being a natural heretic, looked at the data and discovered that he and his colleagues were no more accurate than random guesswork (his predictions were actually being measured against performance at officer school, rather than in combat). Obviously, as the army is the army, it carried on with the tests regardless. You can see Kahneman’s comments on the subject on YouTube (LINK – comments on officer selection starts at 10:00, though the entire video is well worth watching). He calls this phenomenon “the illusion of validity”.

His conclusion, based on decades of research into the predictive performance of experts, is that a more effective system would be to conduct highly structured interviews, with each candidate asked the exact same, very simple questions. The success of officer candidates is then regressed onto the answers to these questions. In short order, the simple equation that results will be more predictive of the army experts running around AOSB with clipboards.

Photo of AOSB used under OGL.