Fixed-wing aircraft stall detection using AI/ML

Authors: Emma Haag, Jesse Heminger, Lokesh Merupula, Nav An, Saravana Vallaban, Tewodros Tamene.

Sponsor: Dr. Lance Sherry, Associate Professor, Director of the Center for Air Transportation Systems Research at the Volgenau School of Engineering at George Mason University

Instructors: Prof. Brett Berlin & Prof. Bernard Schmidt

An aerodynamic stall is a hazardous condition for fixed-wing aircraft during which the airflow over the wings necessary to generate sufficient lift for flight becomes turbulent. The turbulence reduces and abruptly ends the lift necessary to maintain flight, and the aircraft subsequently plummets from the sky. The time between the initial onset and realization of stall ranges from fifteen to twenty seconds, leaving minimal time for pilots to implement necessary life-saving aircraft maneuvers. Fixed-wing aircraft are often equipped with several preventative or detective controls, including a minimum recommended operating speed above potential stall speed, as well as a haptic alert issued through the physical pilot controls called a stick shaker. Despite these preventative and detective controls, there is a history of catastrophic flight accidents attributed to stalls due to controls failure.

This project attempts to predict the stall as early as possible using Machine Learning and AI techniques. Along with prediction and generating a warning, this project attempts to explain what contributed to the warning using the explainable ML package LIME. With the lack of publicly available data for this case study, we used simulated data for training prediction algorithms. The simulation follows the aerodynamic equations provided by our sponsor Dr. Lance Sherry.

The primary objectives of this project are
1. Simulate necessary flight data, at least 10,000 flights simulated.
2. Explore feature engineering and selection.
3. Use Logistic Regression, XGBoost, and LSTM models to train and predict
4. Generate comparisons or accuracy, recall, and latency
5. Eplainability using LIME

The Simulation

Using the aerodynamic equations provided by Dr. Lance Sherry code to simulate flight data has been developed which includes identifying stall, uncommanded descent, and uncommanded descent high and uncommanded roll. Our take on this as a classification problem tries to identify the above 3 conditions given the data until that point in time. The below diagram shows the high-level design of the simulation.

Simulation high-level flow chart

Given the set of initial conditions generated randomly for each simulation, the attributes of flight change within simulation according to the aerodynamic equations. Each simulation uses a step of 0.1 seconds and ends 10 seconds after the uncommanded descent high and uncommanded roll condition is detected. The simulation identifies the onset of the stall when these conditions are met starting from a change in altitude. The below figure shows the results of a simulation with a red vertical line identifying the onset of stall, uncommanded descent, and uncommanded descent high and uncommanded roll. These are the target variables in the prediction/classification models. The definitions are:

1. The onset of stall: at the time to buffet
2. Uncommanded descent: onset of stall and increasing positive AOA, and decreasing airspeed
3. Uncommanded descent high, uncommanded roll: all the above and decreasing vertical speed and Std. of altitude > 7

Single simulation strip chart with classes marked in red

Feature engineering and feature selection

Rolling mean, rolling variance, exponential rolling mean, and exponential rolling variance of the below flight attributes were generated.
• Pitch
• Flight Path Angle
• Airspeed
• AoA (Angle of Attack)
• Roll
• Vertical Speed
• Altitude

A simple Random Forest classification model with parameters of n_estimators=100 and random_state=1 was developed for this purpose. The model took a few hours to run and an accuracy of 87% was obtained. The following is the resulting score of each dependent variable in the data in predicting stall. Angel of attack, pitch roll, vertical speed, airspeed, and altitude are the important features in that order.

feature importance with random forest

Modeling

Logistic Regression

A set of logistic regression models is developed to predict the varying stages of stall — onset, uncommanded descent, etc. These models leverage three different sets of input variables: 1) raw input variables without feature engineering; 2) rolling window statistics; 3) a combination of raw input and rolling window statistics. Note that the predictions for the earlier stages of stall will feed into models predicting the subsequent stages of stall.

Feedforward logistic regression architecture

Incorporated time element using rolling means and variances calculated with varying time windows for each target variable (e.g., the rolling variance of Angle-of-Attack). Features used were “Raw” input variables (without transformation): pitch; flight path angle; airspeed; etc., Individually tuned “Rolling Window” input variables, and 10 fold cross-validation was used. Three binary classification models were developed to allow for individual tuning.

Accuracy and Recall of Logistic Regression

Single Flight Simulation — Predicted vs. Actual

XGBoost

Since the XGBoost model is not a time-series model, lag features, rate of change features, and time window segmentation are used to make it suitable for a time series model development. Mainly, Python’s XGBoostClassifier library is used for model development. Additionally, since the dataset is large and required an advanced computing mechanism, the Argo cluster was utilized for the model development and related analysis.
Various feature engineering techniques were tried as per below:
• Lag features
• Rate of Change features
• Time Window Segmentation
Lag Features:
Dataset was split into training and test datasets. Lag features were generated for the train/test sets. A total of 65 features, including ~55 lag features were input into the XGBoost model. The performance was evaluated using the F1-Score. Since this is an imbalanced dataset, the F1-Score is a better metric for the evaluation of this model.

• Rate of Change Features: The rate of change at various window spans was calculated. The best combination was determined.
• Time Window Segmentation: The time series data was segmented into various window spans.
• Summary statistics such as mean, variance, etc., were calculated for each window.

Confusion matrix with the different set of features

Using random search to obtain hyper-parameters and 10 fold cross-validation along with feature engineering and selection algorithms, XGBoost was able to attain a good recall score overall.

Observed vs Predicted/Classified

LSTM

For LSTM feature engineering, the sliding window technique is used. A prediction model with more than one time variable to predict the next step is the sliding window model. For example, the value at t and the value at t+1 is used to predict the value at time t+2. The models can be developed using the current time t and previous times t-1 as input variables to predict t+1. In this technique, a lookback window of a certain time is chosen, and the class is predicted looking forward to a certain data point. The shift of certain data points is also chosen in this technique.

LSTM model was built with tf.keras.Sequential model and uses 29 hidden layers. Sigmoid is used as the activation layer with a dense layer of 1 unit. Three experiments are conducted based on the different sliding windows -

• Back window = 10,Forward Window =1,predict class 1
• Back window 10, forward window = 200, predict class 3
• Back window 10, forward window = 200, predict window (Class 1,2,3 or class 3)

Accuracy and Recall with LSTM experiments

Explainability

Explainability is essential with any AI/ML model which includes Simulatability, Decomposability, and Transparency. Blackbox tools like LIME help understand contributing features.

XGBoost predictions explained with LIME

Logistic Regression predictions explained by the features using LIME

Summary

52.4 sec into the flight journey, the model(back window =1 sec and forward window = 0.1) is able to predict the onset of stall at 52.4 sec. And at 51.3 sec into the flight journey, the trained model (back window = 1 sec and forward window = 20 sec ) is able to predict uncommanded descent high( class 3) would happen at 71.3 seconds. XGBoost and Logistic Regression predictions are shown in the graph below for the three classes.

Models prediction vs observed strip chart

Results comparison

• XGBoost and Logistic Regression could achieve high accuracy, recall rate, and efficient latency with effective feature engineering and selection techniques.
• LSTM could predict all three conditions ahead of time but with latency.
• Explainability is essential with any AI/ML model which includes Simulatability, Decomposability, and Transparency. Blackbox tools like LIME help understand contributing features.

Enhancements for future

• Better hyper-parameter tuning, adding more layers, and dropout layer will boost LSTM performance significantly.
• Simulation can be improved by unsupervised models and/or anomaly detection algorithms.
• Interface/dashboard to predict and show explainable features.
• Two sets of simulations/algorithms running for each side of the aircraft and a summarizing model.