Leveraging multimodal data from a designed experiment of pilots landing a simulated aircraft, we developed a machine learning pipeline for classifying flight difficulty, called the Multi-Modal Functional-based Decision Support System (MMF-DSS). MMF-DSS distills a tabular set of features from our multimodal and functional physiological data through the use of functional principal component analysis, summary statistics, and BorutaSHAP. In this manner, information is derived from the time-series data via the generation of hundreds of features, of which a small subset having the most predictive capability is discerned. Four full factorial designs are used to perform hyperparameter tuning on a set of classifiers. In so doing, a superlative technique is identified. Impacts on executive decision making are examined as well as associated policy making implications. Alternative classifiers are considered for use within our pipeline that trade predictive accuracy for cost efficiency, and recommendations for choosing among these alternatives is provided.
MMF-DSS provides an innovative, interpretable framework to process and classify multimodal functional signals by using (1) functional principal component analysis (FPCA) and summary statistics to distill the complex information into a tabular set of features, (2) BorutaSHAP to select a small subset of features having the most predictive capability, and (3) a flexible framework to choose from a variety of ML algorithms to make the most accurate classification.
The United States Air Force (USAF) has struggled to train enough pilots to meet operational requirements. Technology has advanced rapidly over the last 70 years but USAF pilot training has not. Modern operational requirements demand a change and, for this reason, USAF senior leadership has advocated for automation of instructor and evaluator pilots in various forms to minimize pilot training bottlenecks. Therefore, given this need and the costly nature of purchasing new equipment, MMF-DSS focuses on this classification task using cost-effective physiological-based signals to build the classification model.
Caballero, W. N., Gaw, N., Jenkins, P. R., Johnstone, C. (In Review) Toward Automated Instructor Pilots in Legacy Air Force Systems: Physiology-based Flight Difficulty Classification via Machine Learning, Expert Systems with Applications. https://dx.doi.org/10.2139/ssrn.4170114
