2020: The Year the AAR/IP Finally Dies.

Sherilyn Burris
Jan 1 · 13 min read

Using Design to Break the Cycle of Disaster > Learn > Disaster > Repeat.

After Action Reports and Improvement Plans and lessons observed and mistakes repeated. This is the decade we change.
After Action Reports and Improvement Plans and lessons observed and mistakes repeated. This is the decade we change.

__

I’ve got a degree in History (represent!) so naturally I read historical documentation when I’m not working. I’ve collected a few hundred delightfully dated PDFs on every post-industrial disaster in the USA and the notable international disasters that shaped our modern world. I read them voraciously to absorb the gems of wisdom that have lived on dark and quiet government shelves for decades. It’s not a shock to any practicing emergency manager that there’s a consistent theme: the entire section on what went wrong and why has been copied and pasted for the past hundred disasters without ever getting better.

Emergency managers write extensive reports on storms, to find out what we did right and what we did wrong. The outline is consistent, developed originally in the Department of Homeland Security’s Homeland Security Exercise and Evaluation Program (HSEEP) though, as stated in their documentation, intended for exercises (that’s jargon for disaster practice) and not actual disasters. More on that later.

Here are a few examples:

1. Expecting the Unexpected. If there was one overarching lesson, it was to expect the unexpected. Common assumptions can lead to serious problems, particularly when they involve predicting where a hurricane will make landfall.

2. Dealing With Inexperience. “Most of us were inexperienced,” the leadership said. “The State had not been hit by a major hurricane in years. In our office, there were just a few people who had worked through a hurricane. Only two of the staff had hands-on emergency response experience.”

3. Clearing the Debris. A hurricane is a perfect machine for creating debris, as it destroys and scatters the remains of structures, signs, utility poles, and trees. It can take months to clear an area of storm debris completely. There were also practical lessons to be learned about how to administer a debris removal contract. “Debris removal was a challenge to manage,” … ”Contractors who were paid by weight were sometimes inclined to pick up only the heavy debris and leave the lighter limbs and rubbish behind — just as those paid by volume might tend to ignore the heavier debris. Agencies needed to inspect and track the process.”

And here is another improvement from the National Weather Service. They say,

“New demands for information are placed on the NWS due to increasing and changing societal vulnerability to weather, growing awareness of this vulnerability, and technological advances, especially in computing and communications. These demands, and the changes brought about by the ongoing NWS restructuring, continue to impact and influence both the current operations and planning for future operations of the NWS.”

Those first three were lessons from the Florida Division of the Federal Highway Administration following the 2004 hurricane season, despite being almost identical to many 2017 (and 2018) hurricane season lessons. The Weather Service comment was from 1994. 1994. What this tells you is that we haven’t been able to actually learn these lessons in more than 25 years.

Think these are isolated examples? Play along and test how much you know about a few domestic disasters. Can you identify the state, the hazard, and the date?

  1. On some occasions during the response, confusion about who was at risk of developing the disease and ambiguities about the extent of public health officials’ authority resulted in public health actions being influenced by political pressures.
  2. Need for quick up-to-date disaster related information. The incident was so large and developed so quickly that information resources were stretched too far. Public and legislative representatives should have a better understanding of State and Local Fire Agencies’ need to adhere to safety standards and procedures. Local government should work on planning on providing mental health services and the particular needs of the elderly and disabled. Some of these issues are also state and federal concerns.
  3. Accountability of deployed personnel was dangerously lacking. Team members were frequently, and sometimes repeatedly, relocated to meet the demands of a dynamic situation. It was often impossible for managers from Assisting States to locate and contact their deployed personnel. Deployed personnel sometimes arrived with inadequate equipment and inappropriate gear for the primitive conditions in the stricken area. In such cases, they added to the burden of the Requesting States, requiring much of the same support as the victims. A great deal of time was consumed in trying to precisely define the desired attributes of required resources on the one hand and the qualifications of available assets on the other. This illustrates the need for an acceptable form of “resource typing,” a requirement identified by representatives of both the Requesting and Assisting States.
  4. All phones lines were tied up by people telling us there had been a disaster. Communications systems have grown over the years in sophistication, capability and capacity. Emergency conditions such as these stress such systems to their limit, at times exceeding those limits. Provision for the additional safety of un-interruptible power supplies for computers and communications equipment was listed several times as a priority.
  5. State and local governments have little fiscal stake in floodplain management; without this stake, few incentives exist for them to be fully involved in floodplain management. State governments must assist local governments in dealing with federal programs but, in many cases, do not become involved in federal-local activities. Federal and state oversight over non-federally constructed levees was and remains diffuse. Several states regulated construction in floodplains, but many did not. The situation was further exacerbated by the potential for future flow increases that could occur if development continued upstream and by the uncertainty about changes that may occur in long term weather patterns. Few states controlled either the decision about where levees are placed relative to the river channel or whether a particular levee should be protected from overtopping (floodfought) during a flood, although such actions can have hydraulic and environmental consequences elsewhere. Some states had little or no involvement in the processes associated with federal levee programs since federal agencies generally dealt directly with the local organizations responsible for levee operation and maintenance.
  6. Preparedness problems were linked to weaknesses in state and local programs, as well as in FEMA’S assistance and overall guidance. These problems included inadequate planning and training for recovery, low participation by elected officials in training and exercises, inadequate or no standard operating procedures for response and recovery activities, and inadequate coordination between several federal agencies. These inefficiencies resulted from staffing and coordination difficulties between agencies at all levels. State and federal agencies, including FEMA did not manage their recovery activities as efficiently as possible. This resulted in delays in providing assistance and in duplicate payments for certain activities. Also, legislation may be needed to clarify FEMA’S role in responding to disaster-related, long-term housing needs.
  7. Faster response is needed than what occurred. The initial grid lock among local, county, state, and federal governments delayed massive response 2–3 days. In the case of a catastrophic earthquake, many lives could have been lost while awaiting aid. We need provisions for an automatic response to a catastrophic disaster. Based on pre-established criteria, we would begin automatically responding until directed otherwise. For example, any category 4 or 5 hurricane hitting a populous area or any 7.0 earthquake in a populous area would trigger an automatic federal and DOD response. We learned that contracting should commence immediately for those items that Federal agencies could not provide or supply in the quantities required, i.e., reefer vans, ice, dumpsters, and porto-lets. Critical factors to be considered for initial logistics operations for disaster relief are the availability, locations, and selection of aerial ports of debarkation, their distance from the relief area, storage capability, and the logistics infrastructure required to move incoming supplies.
  8. At annual workshops the National Hurricane Center (NHC) should continue to emphasize to emergency managers current forecasting capabilities and limitations. NHC should include in its advisories underlying reasons for significant forecast changes. Understanding what the forecast means and reasons underlying forecast changes would increase the confidence of emergency managers and the media in the advisories used by NHC, thereby, enhancing vital cooperation between local National Weather Service offices, local officials and the media. It also would heighten public awareness to changes which require additional public response.
  9. To strengthen the international response, we recommend that the Secretary of Health and Human Services, in collaboration with the Secretary of State, work with the World Health Organization (WHO) and official representatives from other WHO member states to strengthen WHO’s global infectious disease network capacity to respond to disease outbreaks, for example, by expanding the available pool of public health experts. Second, to help Health and Human Services prevent the introduction, transmission, or spread of infectious diseases into the United States, we recommend that the Secretary of HHS complete the necessary steps to ensure that the agency can obtain passenger contact information in a timely and comprehensive manner, including, if necessary, the promulgation of regulations specifically for this purpose.
  10. Personnel and vehicle congestion at the Emergency Operations Center (EOC) will be more effectively checked and managed in future large scale activations. Some persons appeared regularly and unannounced at the facility, but did not have a discernible and official function. This caused multiple minor issues with the limited space and increased noise and will be controlled more with stricter sign-in procedures and special access badge entry required to gain access past property guards. There is a need for better documentation of actions taken, tasks remaining, materials used, and other important data that would improve the effectiveness of how Planning functions in its GIS role. This information is useful to orient staff beginning a new shift and is also valuable for tracking and reporting purposes. GIS staff would benefit from a clear understanding of what needs to be recorded in WebEOC, and where to document additional facts and instructions (contact information lists, technical instructions, etc.).
  11. Existing federal programs designed to protect and enhance the floodplain and watershed environment are not as effective as they should be. They lack support, flexibility and funding, and are not well coordinated. As a result, progress in habitat improvement is slow. The nation needs a coordinated strategy for effective management of the water resources of the upper Mississippi River Basin. Responsibility for integrated navigation, flood damage reduction and ecosystem management is divided among several federal programs. To reduce the vulnerability to flood damages of those in the floodplain, the Administration should give full consideration to all possible alternatives for vulnerability reduction, including permanent evacuation of floodprone areas, flood warning, floodproofing of structures remaining in the floodplain, creation of additional natural and artificial storage, and adequately sized and maintained levees and other structures.
  12. This report is dedicated to the many people with disabilities and activity limitations who lost their independence or their lives because information transfer and the lessons learned and documented over the last 30 years, are not yet uniformly applied. … The term “special needs” does not work because it does not provide guidance to operationalize needed planning tasks. A better way to think about the needs of people with disabilities and activity limitations is to use an orientation that considers major functional needs: communication, medical, maintaining functional independence, supervision, and transportation. … State emergency planners should develop and offer guidance to local government regarding how to actively recruit qualified people with a variety of disabilities (i.e., mobility, vision, hearing, cognitive, psychiatric, and other disabilities), and how to involve organizations with expertise on disability issues in all phases of emergency management planning.

What’s going wrong here?

We’re using a document meant to track controlled, intentionally developed, specific disaster exercise outcomes and applying it to an uncontrolled, unspecific and otherwise completely unplanned hazardous event that led to a disaster. Exercises are written and executed to test certain capacities, and the AAR/IP forms track whether the exercise objectives were met ― whether the capacity is acceptable against the hypothetical scenario presented.

It’s a spreadsheet:

  • Core Capability
  • Issue/Area for Improvement
  • Corrective Action
  • Primary Responsible Organization and Contact Information
  • Start Date, Completion Date

Spreadsheets don’t solve problems. Writing problems down is a start ― and that’s all it is: a start. But the AAR/IP document is considered the gold standard, the ultimate achievement, the end all, be all, soup to nuts, one and done.

In the text, agencies are instructed to provide a root cause analysis or summary of why the full capability level was not achieved, starting with a clear observation statement of the problem or gap. There is little instruction on what this means or how to do it. The accompanying HSEEP in-person course is 8 to 24 hours long and focuses mostly on exercise design.

But let’s back up. The hypothetical exercise objectives to test certain capabilities are written into the exercise with intent. But disasters don’t work that way. No earthquake shows up with a checklist of what it’s going to test or how it’s going to test it.

If the AAR/IP process was working, we wouldn’t be repeating the exact same issues over and over again. It’s a system problem. We must take these evaluations about “what happened” to being a “Consequence Reduction System” activity ― an ongoing, iterative system ― not a document and not a report.

How?

There are two industries already doing evidence tested, data-driven activities that create remarkable improvements ― the healthcare industry and the design industry.

Quality improvement in healthcare stems from over a century of quality management and pairs nicely with the International Organisation for Standardization’s ISO 9000, Quality Management, first published in the 1980s. Healthcare industries began implementing quality processes as a formalized approach in the 1990s with innovative publications like The Improvement Guide: A Practical Approach to Enhancing Organizational Performance and To Err Is Human: Building a Safer Health System.

One of the best use cases is the Model for Improvement, or MFI, developed by the Institute for Healthcare Improvement (IHI) in 1996. This improvement system was well in use by the time HSEEP rolled out after the September 11, 2001 terrorist attacks. The MFI asks:

  • What are we trying to accomplish?
  • What changes can we make that will result in improvement?
  • How will we know that a change is an improvement?

And already, the MFI is asking an extraordinarily difficult question that emergency managers completely ignore when approaching post-disaster incident reports and improvement planning. How do we know that our proposed solution actually fixed the problem?

If the recommendations are being implemented (and implemented correctly and in a sufficiently timely manner), why aren’t they working to stop the spiraling repetition of the same “lessons” disaster after disaster? Because the potential solutions aren’t being tested before they’re implemented.

And that’s where design comes in.

Design sprints are popular and effective ways to solve problems quickly. Sprints don’t create bandaids or distractions away from the problems in the ways some haphazard post-disaster reports do because sprints aren’t about completing a fluffy paper report.

A typical AAR/IP is completed within 90 days of an incident but the actual improvement planning content of the document displays as a wish list, with dates “to be completed” filled in with arbitrary but good intentions. The reports include references to activities “we think this will solve the problem.”

Instead of the AAR/IP using a typical table of fill-in-the-blank form, design uses variable and unrestricted user input. Sprints capture freeform ideas and refines them into testable and usable solutions without the goal being to properly fill out a government form.

Design sprints go a few steps further to actually test the proposed solutions before committing years of time and thousands of dollars of resources. Sprints reduce the AAR/IP process time down from one year or more to one week. Using design, we separate the required documentation (what happened in the disaster, the “after action” part of the report) from the improvement planning (IP) pieces.

Not having the AAR/IP prototype and test is why the historically gold standard procedures fail. All the effort is expended when the solution was never correct to begin with. There was no formal problem statement, no root cause analysis, and the improvements, written as goals and not achievements, become stabs in the dark and grasping at straws.

While the design sprints keep going, the AAR/IP don’t have to be an entirely separate process. The week-long sprint is well within the 90-day typical documentation production timeline. Potential solutions can be designed, tested, and refined before being launched into the final document and written into future policies and procedures.


After Action Reports and Improvement Plans (AAR/IP) have included the exact same lessons learned for hundreds of disasters and thousands of exercises. They’re lessons observed and mistakes repeated again and again. This is the decade we switch the narrative.


Answers

I picked these particular “lessons” because I personally experienced them in my work, despite not working for these agencies or on these events. I threw in a few obscure items out of sheer enthusiasm for the prolific misbeliefs on the emergency operations center being someone’s own personal social club/restaurant/hotel/shelter and epic frustrating simple things like forgetting to include contact information for someone requesting a resource (see #9).

Some of these examples were obvious for the place or hazard but you may have gotten caught up on the year. If you said Hurricane Maria, Hurricane Michael, Hurricane Katrina, or Ebola, you weren’t altogether wrong. These lessons learned are also listed in those After Action Reports, as well.

  1. National; Anthrax; 2001. From: Anthrax 2001: Observations on the Medical and Public Health Response. Gursky, E., Inglesby, T.V., and O’Toole, T. Biosecurity and Bioterrorism: Biodefense Strategy, Practice, and Science. Volume 1, Number 2, 2003.
  2. California; Wildfires; 2003. From: 2003 Southern California Fires After Action Report, June 17, 2004.
  3. Florida; Hurricane Season; 2004. From: Emergency Management Assistance Compact (EMAC) After-Action Report for the 2004 Hurricane Response.
  4. California; Loma Prieta Earthquake; 1989. From: State of California Law Enforcement Operations Report: Loma Prieta Earthquake, published May 1990.
  5. Midwest; Flooding; 1993. From: Learning from the Mississippi Flood of 1993: Impacts, Management Issues, and Areas for Research by Gerald E. Galloway, Jr., Hydrometeorology, Impacts, and Management of Extreme Floods, November 1995.
  6. National; Hurricane Hugo and the Loma Prieta earthquake in September and October 1989; published in 1991. From: Disaster Assistance: Federal, State, and Local Responses to Natural Disasters Need Improvement. GAO/WED-9143 FEMA’s Response to Natural Disasters. March 1991.
  7. Florida; Hurricane Andrew; 1991. From: Forces Command Joint Task Force Andrew After Action Report.
  8. Puerto Rico, Carolinas; Hurricane Hugo; 1989. From: Service Assessment Report Hurricane Hugo September 10–22, 1989; Office of Climate, Water, and Weather Services (OCWWS), May 1990.
  9. National; SARS; 2003. From: Asian SARS Outbreak Challenged International and National Responses, GAO-04–564, published April 2004.
  10. Tennessee; Flooding; 2010. From: Metropolitan Government of Nashville, Tennessee; Severe Flooding, May 2010, Disaster Declaration #FEMA-1909-DR, After Action Report/Improvement Plan.
  11. National; 1994. From: Sharing the Challenge: Floodplain Management into the 21st Century. A Report of the Interagency Floodplain Management Review Committee, published June 1994.
  12. Southern California Wildfires After Action Report, 2008. Access to Readiness Coalition, Center for Disability Issues and the Health Professions.

Sherilyn Burris

Written by

Disaster dork. Crisis master of ceremonies. Public health nerd. I write, drive Minis, and love the Oxford comma.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade