re-Evaluating the Value of Military Exercises

The immediate need for evidence based training and exercise programs in the U.S. Military.

(U.S. Air Force photo by Airman 1st Class Christopher Callaway)

The views expressed are my own and do not reflect the official policy or position of the Joint Staff, U.S. Army, Department of Defense, or the U.S. Government.

Recently the Chairman of the Joint Chiefs of Staff highlighted the importance of the joint exercise program. A program that costs tax payers around $650 million annually. The program trains combatant commanders, their staffs, and supporting military forces to respond to operational plans and contingencies around the world.

We are reenergizing and reorienting the joint exercise program to develop a shared understanding about the array of threats we face, collectively assess our preparedness to respond to contingencies, and uncover vulnerabilities in our operational plans and concepts.

Leaders are quick to offer their vision of the future, even sometimes framing a roadmap for how to get there. Few leaders are truly masters at following through and monitoring progress toward that vision. Even fewer are willing to put their cherished beliefs and critical assumptions up to scrutiny and criticism.

The cost of unevaluated assumptions and underlying belief systems can be devastating.

One of the reason so few leaders actually examine these beliefs is confirmation bias. Nobel prize winning economist Daniel Kahneman explains that “confirmation bias comes from when you have an interpretation, and you adopt it, and then, top down, you force everything to fit that interpretation.”

Having an interpretation of allows us to make sense of our actions; however given how susceptible we are to confirmation bias, leaders should not trust everything they think.

What belief systems underlie the need to reenergize and reorient military exercises? How would the Chairman know if his efforts have been successful? And ultimately, how would one know if any added effort to the joint exercise program contributed to the U.S. military’s competitive advantage over its adversaries?

A strong belief in the power of training permeates military culture — training has become synonymous with readiness.

Rather than accepting this belief in training as dogma. Leaders must look for evidence that military exercise programs contribute to the competitive advantage of the joint force. Commanders currently provide these assessments, usually through after action reports, quarterly reports, and assessment of the impact an exercise had on their unit. But is this really any proof?

Individual stories about the success of an exercise do not amount to evidence for the central claim. Testing these claims is not simple, but it is absolutely critical to know what things work and why they work.

Dr. Richard Cabot — The Association for Clinical Pastoral Eduction

Dr. Richard Cabot is an inspirational leader in this regard. An American physician and pioneer in social work, his study of criminal behavior youth offers a warning for leaders who are content to continue to act without evaluating the consequences of their actions.

The story of Dr. Cabot’s Cambridge Somerville Youth Study was recently featured on Freakonomics Radio.

Dr. Cabot wanted evidence to support his beliefs that a mentoring program could reduce recidivism and criminal behavior. As the first recognized randomized experiment of a social program, Dr. Cabot’s longitudinal study identified 250 at risk kids and randomly assigned them to a control group or a treatment group. The treatment took a broad range of social work interventions. Over a period of 30 years, researchers were able to monitor and measure the effect of the treatments toward their desired outcomes.

Surprisingly in 1948 and in the 50’s, they found a null effect. There was no difference between the treatment group and the control group on criminal offenses.

Aided by advances in computation, researchers were later able to perform more sophisticated analysis of the data along seven outcome measures, and the results of the analysis were completely surprising. In addition to criminal offenses, the other six outcome measures were:

  • Life expectancy
  • Mental health measures
  • Physical Health measures
  • Alcoholism
  • Job satisfaction
  • Marriage satisfaction
On all seven measures, the treatment group did statistically, significantly worse off than the control group.

The results of this study should be cause for both concern and optimism. Although the original treatments had the opposite of the intended effects, there is evidence that programs can change behavior.

Department of Defense Photo by Navy Petty Officer 2nd Class Dominique A. Pineiro

Dr. McCord, one of the primary researchers in this study, made significant contributions to the field of criminology by empirically challenging her own long held beliefs about social programs. Two of her observations about the study are applicable more broadly, and should serve to inform military leaders and policy makers.

  1. Nothing provides better evidence to assess the validity of cherished beliefs than using random assignment.
  2. Carefully collecting records can provide valuable evidence later, even after the original study might be over.

Policy makers and military leaders should demand evidence to support their beliefs. If the reason for this is not self-evident, Richard Feynman may have illustrated it best.

The first principle is that you must not fool yourself — and you are the easiest person to fool.

The purpose for any change in military exercise programs should be to create and maintain a competitive advantage for the U.S. military. However, to what extent and what particular components should be changed must be decided by the evidence. Untested assumptions, dogma, or charismatic personalities should not be the basis for changes to the system.

Leaders who solely trust their intuition and ignore the evidence of real world feedback do so at their peril.