Tool Time
Beta-Testing a Maintenance Safety Culture Assessment Toolkit
by Kylie Key, Ph.D., Inchul Choi, Ph.D., and Justin Durham, FAA Civil Aerospace Medical Institute
There are benefits to having a positive safety culture — but how do you know whether you have an adequate safety culture, and how to improve? Helping the aviation industry answer these questions is a goal of researchers at the FAA’s Civil Aerospace Medical Institute (CAMI).
What is a Safety Culture?
Safety culture can be defined as employees’ perceptions of how much safety is valued in their organization and the extent to which risk-taking behaviors are viewed as necessary to ensure timely completion of tasks. The level of safety culture in the workplace is typically measured with a survey of employees’ commitment to safety. But it’s not just an employee’s commitment to safety — after all, employees don’t work in a vacuum. Safety culture is shaped by the work environment. While there are many surveys available, they can be expensive and may require trained researchers or consultants to analyze and interpret the data, making them out of reach for smaller organizations. Major airlines may be able to afford these costs, but smaller organizations usually cannot.
FAA Maintenance Safety Culture Assessment Toolkit
Researchers at CAMI are working to provide a stand-alone process for organizations to have full ownership of their culture assessment and associated proprietary data. The FAA Maintenance Safety Culture Assessment Toolkit is designed to include everything needed for a DIY culture assessment and improvement effort that will allow organizations control of the process by providing the survey instrument, data analysis tools, and educational/guidance materials for safety culture promotion.
This toolkit goes beyond simple measurement of employee commitment to safety, which is the focus of most culture surveys. It measures environmental factors from the work environment, such as the job resources available and the demands that are faced to complete a task. Together, job resources and demands form the foundation of culture, in turn influencing an employee’s outcomes (e.g., job satisfaction, risk-taking behavior) and the organization’s bottom line (e.g., errors, accidents/injuries, productivity).
Whether you have a workforce of two or two thousand, job resources and demands are drivers of safety culture and should be included in any culture assessment. Although the toolkit is currently in development, we urge you to honestly assess yourself and your organization using sample items in Table 1, considering ways that you may be able to allocate additional job resources and reduce demands.
As noted, the toolkit is under development and the instrument is being refined. The current version has about 180 items, but takes only 20–30 minutes to complete and is administered via an anonymous online survey link. The data analysis tool plots the distribution of participant responses, along with the goal set by the organization, to pinpoint opportunities for improvement.
Lessons Learned from Beta-Testing
Before publicly releasing this toolkit, we are beta-testing to ensure it meets the needs of the aviation industry. So far, we are in various beta-testing stages with three organizations, each with 50-plus employees: a large part 145 maintenance operator, a rotorcraft maintenance organization, and a group of pilots. Some key lessons learned and next steps are:
- Each organization faces different operational challenges, necessitating tailored survey content. A future goal is to empower users to tailor the survey to their operations without needing our help.
- Some organizations cannot spare 20 minutes for every employee to complete the full survey, but they still want a quick pulse of their culture. A future goal is to create a “short form” of the survey, perhaps by expanding Table 1. This short form would include only the top safety culture items that apply to every organization, large or small.
- Without ensured anonymity, no amount of incentives or advertising for the survey will result in an adequate participation rate. Protecting anonymity is key, no matter how many (or few) respondents there are.
The aviation industry has an appetite for the toolkit and has encouraged us to expand our efforts to other operations types (e.g., cabin crew, ground handling). We have also been asked to make a scaled-down version of the toolkit for smaller operations such as general aviation (GA).
The most critical challenge to a successful GA toolkit is protecting employee anonymity to ensure an adequate participation rate and sincere responses.
Culture Assessment in GA
The most critical challenge to a successful GA toolkit is protecting employee anonymity to ensure an adequate participation rate and sincere responses. Our tentative plan is to: a) create a customizable anonymous survey administered online, so that responses are de-identifiable, b) prevent managers from seeing individual responses by aggregating the data into a centralized database, and c) create an automatically generated report with an overall picture of the organization’s safety culture, but not disclose detailed information that could identify participants (such as job role or scheduled shift).
Additionally, if the number of the respondents is very small, we could average the data across multiple similar organizations with similar size and scope of operations. Each organization could then receive the same report about industry on average, revealing general trends and plausible issues that they may have.
This is one potential method for making the toolkit work for GA, but we need smaller organizations to provide expert feedback to help us improve the survey and the process. We need to better understand the unique operational challenges in the GA environment, what we are missing, and how we can make a better tool for smaller organizations. To make this toolkit work — we need you. If you are interested in testing the survey or have suggestions, please contact Dr. Kylie Key at kylie.n.key@faa.gov. We continually look for ways to improve and look forward to hearing your feedback!
The authors thank the research funding source, the FAA’s Human Factors Division (ANG-C1), and collaborators Dr. Bill Johnson, Dr. David J. Schroeder, and Blake L. Nesmith for their contributions to this project.
Learn More
Kylie Key, Ph.D. is an Engineering Research Psychologist for the Flight Deck Human Factors Research Laboratory at the FAA’s Civil Aerospace Medical Institute (CAMI).
Inchul Choi, Ph.D. is a Sr. Statistician and Human Factors Engineer for Cherokee Nation 3S at the FAA’s CAMI. He performs data analysis for flight deck and control tower research projects.
Justin Durham, M.A. is a Human Factors Researcher for Cherokee Nation 3S at the FAA’s CAMI. He is currently pursuing a Ph.D. in Cognitive Psychology at the University of Oklahoma.
