Image for post
Image for post

Tool Time

Beta-Testing a Maintenance Safety Culture Assessment Toolkit

FAA Safety Briefing
Jul 1, 2020 · 6 min read

by Kylie Key, Ph.D., Inchul Choi, Ph.D., and Justin Durham, FAA Civil Aerospace Medical Institute

There are benefits to having a positive safety culture — but how do you know whether you have an adequate safety culture, and how to improve? Helping the aviation industry answer these questions is a goal of researchers at the FAA’s Civil Aerospace Medical Institute (CAMI).

A mechanic at work.
A mechanic at work.

What is a Safety Culture?

Safety culture can be defined as employees’ perceptions of how much safety is valued in their organization and the extent to which risk-taking behaviors are viewed as necessary to ensure timely completion of tasks. The level of safety culture in the workplace is typically measured with a survey of employees’ commitment to safety. But it’s not just an employee’s commitment to safety — after all, employees don’t work in a vacuum. Safety culture is shaped by the work environment. While there are many surveys available, they can be expensive and may require trained researchers or consultants to analyze and interpret the data, making them out of reach for smaller organizations. Major airlines may be able to afford these costs, but smaller organizations usually cannot.

FAA Maintenance Safety Culture Assessment Toolkit

Aviation Human Factors graphic
Aviation Human Factors graphic

Researchers at CAMI are working to provide a stand-alone process for organizations to have full ownership of their culture assessment and associated proprietary data. The FAA Maintenance Safety Culture Assessment Toolkit is designed to include everything needed for a DIY culture assessment and improvement effort that will allow organizations control of the process by providing the survey instrument, data analysis tools, and educational/guidance materials for safety culture promotion.

This toolkit goes beyond simple measurement of employee commitment to safety, which is the focus of most culture surveys. It measures environmental factors from the work environment, such as the job resources available and the demands that are faced to complete a task. Together, job resources and demands form the foundation of culture, in turn influencing an employee’s outcomes (e.g., job satisfaction, risk-taking behavior) and the organization’s bottom line (e.g., errors, accidents/injuries, productivity).

Whether you have a workforce of two or two thousand, job resources and demands are drivers of safety culture and should be included in any culture assessment. Although the toolkit is currently in development, we urge you to honestly assess yourself and your organization using sample items in Table 1, considering ways that you may be able to allocate additional job resources and reduce demands.

Table listing job resources and job demands.
Table listing job resources and job demands.

As noted, the toolkit is under development and the instrument is being refined. The current version has about 180 items, but takes only 20–30 minutes to complete and is administered via an anonymous online survey link. The data analysis tool plots the distribution of participant responses, along with the goal set by the organization, to pinpoint opportunities for improvement.

Lessons Learned from Beta-Testing

Before publicly releasing this toolkit, we are beta-testing to ensure it meets the needs of the aviation industry. So far, we are in various beta-testing stages with three organizations, each with 50-plus employees: a large part 145 maintenance operator, a rotorcraft maintenance organization, and a group of pilots. Some key lessons learned and next steps are:

  1. Each organization faces different operational challenges, necessitating tailored survey content. A future goal is to empower users to tailor the survey to their operations without needing our help.
  2. Some organizations cannot spare 20 minutes for every employee to complete the full survey, but they still want a quick pulse of their culture. A future goal is to create a “short form” of the survey, perhaps by expanding Table 1. This short form would include only the top safety culture items that apply to every organization, large or small.
  3. Without ensured anonymity, no amount of incentives or advertising for the survey will result in an adequate participation rate. Protecting anonymity is key, no matter how many (or few) respondents there are.

The aviation industry has an appetite for the toolkit and has encouraged us to expand our efforts to other operations types (e.g., cabin crew, ground handling). We have also been asked to make a scaled-down version of the toolkit for smaller operations such as general aviation (GA).

The most critical challenge to a successful GA toolkit is protecting employee anonymity to ensure an adequate participation rate and sincere responses.

Culture Assessment in GA

The most critical challenge to a successful GA toolkit is protecting employee anonymity to ensure an adequate participation rate and sincere responses. Our tentative plan is to: a) create a customizable anonymous survey administered online, so that responses are de-identifiable, b) prevent managers from seeing individual responses by aggregating the data into a centralized database, and c) create an automatically generated report with an overall picture of the organization’s safety culture, but not disclose detailed information that could identify participants (such as job role or scheduled shift).

Additionally, if the number of the respondents is very small, we could average the data across multiple similar organizations with similar size and scope of operations. Each organization could then receive the same report about industry on average, revealing general trends and plausible issues that they may have.

Photo of mechanic in a hangar.
Photo of mechanic in a hangar.

This is one potential method for making the toolkit work for GA, but we need smaller organizations to provide expert feedback to help us improve the survey and the process. We need to better understand the unique operational challenges in the GA environment, what we are missing, and how we can make a better tool for smaller organizations. To make this toolkit work — we need you. If you are interested in testing the survey or have suggestions, please contact Dr. Kylie Key at kylie.n.key@faa.gov. We continually look for ways to improve and look forward to hearing your feedback!

The authors thank the research funding source, the FAA’s Human Factors Division (ANG-C1), and collaborators Dr. Bill Johnson, Dr. David J. Schroeder, and Blake L. Nesmith for their contributions to this project.

Learn More

“Safety culture: Where do we stand?” March 2019 FAA Aviation Mx Human Factors Quarterly, page 5 (PDF)

“Safety Culture Promotion in an Example Mx Organization”
June 2019 FAA Aviation Mx Human Factors Quarterly, page 4 (PDF)

Kylie Key, Ph.D. is an Engineering Research Psychologist for the Flight Deck Human Factors Research Laboratory at the FAA’s Civil Aerospace Medical Institute (CAMI).

Inchul Choi, Ph.D. is a Sr. Statistician and Human Factors Engineer for Cherokee Nation 3S at the FAA’s CAMI. He performs data analysis for flight deck and control tower research projects.

Justin Durham, M.A. is a Human Factors Researcher for Cherokee Nation 3S at the FAA’s CAMI. He is currently pursuing a Ph.D. in Cognitive Psychology at the University of Oklahoma.

Image for post
Image for post
This article was originally published in the July/August 2020 issue of FAA Safety Briefing magazine. https://www.faa.gov/news/safety_briefing/
Image for post
Image for post

Cleared for Takeoff

Voices, stories and news from the Federal Aviation Administration

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store