CRD Project

The aim of this study was to develop methods for mental workload (MWL) and stress assessment on data collected in ambulatory conditions. Such methods can help us develop robust psychological models of performance for first responders such as firefighter and police officers. These jobs require quick decision making while being involved in physical activity. Such conditions, with a mix of both physical activity along with mental stress/workload are much more representative of real-life conditions. Previously done studies linking cognitive states with physiological responses in the literature have always focused on data collected in stationary controlled condition. Such data collection ensures good quality data without any confounding effects on the physiological signals. However, models trained on such data perform poorly on data collected in real-world conditions. Apart from collecting three publicly available datasets on a large number of subjects, we also developed methods to better predict cognitive states in the presence of noise and other confounders. A small description of the datasets is provided below:

Dataset 1: Workload Under Activity Conditions (WAUC) dataset [paper]

Data was collected from 48 subjects performing physical activity (PWL) (no, medium and high levels) on a bike or treadmill while simultaneously performing the MATB-II task design to elicit mental workload at different levels (low, high). Each subject performed 6 (3 PWL X 2 MWL) 10 minute sessions. Various sensors were used to collected physiological data during tasks. The collected signals include EEG (8-channel), ECG, respiration (from chest-band) and PPG, GSR and activity (from smartwatch). The data collection can be summarized as:

Proposed methods:

Multi-scale permutation entropy based features:

We proposed multi-scale entropy features that quantify the fractal behavior of the physiological time series. Additionally, combining permutation entropy with the multi-scale algorithm provides robustness to artifacts. Permutation entropy converts the time series in motif series before calculating entropy These motifs reject any amplitude information in the signal while preserving the shape of the time series and are thus robust to various artefacts. We tested several variants of the scaling and entropy calculation methods and found that permutation entropy along with a composite scaling method gives the best performance for HRV based mental workload assessment [1]. When used for EEG time series, these features once again outperformed the benchmark while also showing complementary behavior with the standard EEG feature set for mental workload prediction [2].

Comparison of noise-robust HRV extraction pipeline with traditional methods:

HRV analysis feature extraction pipeline involves QRS complex peak detection in order to create an interbeat interval (RR) series followed by time- and freq- domain feature extraction. When used in real-world data, noise can often deteriorate the performance of the peak detection methods leading to false peaks and outliers. This can in turn degrade the performance of HRV features. In this work, we compared the traditional feature extraction methods for HRV calculation to new proposed modulation spectrum based heart rate estimation. By separating the noise from the HR signal lobes, the modulation spectrum allows for noise-robust HRV calculation. Our proposed pipeline [3] outperformed the traditional approaches in different physical activity levels when predicting mental workload.

Multi-modal approach for mental workload assessment:

Various signal modalities may contain complementary information about cognitive states. In addition to this, the use of multiple signal streams adds noise robustness to the system as even if one of the signals may be corrupted other modalities may make up for it. Such systems have been explored to various degrees for data collected in control environments. In this work [4], we explored multimodal data including EEG, ECG, breathing, body temperature, PPG and GSR for noise robust assessment of mental workload in ambulatory conditions. Furthermore, by making use of an epoch wise voting scheme we were able to add robustness of various artefacts and further boost performance in a subject-independent evaluation setting.

Dataset 2: Physical Activity under StresS (PASS) dataset [paper]

This dataset was very similar to WAUC dataset in terms of study design. Stress in this case was evoked using relaxing and stressful video games and participants only used the bike for modulation physical activity. This dataset was also collected on 48 participants. The data collection can be summarized as:

Proposed methods:

Spectral descriptor and complexity based features:

HRV frequency spectrum can be divided into two separate bands.. These include the high frequency region modulated by parasympathetic activity and the low frequency region influenced by both parasympathetic and sympathetic activity. Previous studies have shown that both these influences can independently impact HRV complexity and also show interaction with one another. We extracted spectral descriptor and complexity features for bandpassed high and low frequency RR series. We show that these feature outperform the commonly used benchmark and complexity features for stress detection

Dataset 3: Stress/MWL of police trainees [paper] :

Physiological data was collected using a chest-strap from 27 participants during a 15 week course training course while they performed shooting range exercises and arrest simulations. Shooting exercises included handling instructions, stance and shooting exercises while arrest simulations consisted of decision making, team work and intervention when a firearm was involved.

Proposed methods:

Fusion of ultra-short HRV and breathing features:

Ultra-short HRV analysis referred to feature extraction and analysis of ECG epochs smaller than 5 mins has gained popularity over the past few years. Shorter epochs (as small as 30s) improve the time-resolution for HRV analysis but have been only assessed on data collected in controlled environments. Additionally, the impact of fusion of ultra-short HRV with other modalities such as breathing has not been studied. We tested various HRV epoch sizes and fusion with breathing features on in-the-wild police training data. We show that for real world stress and mental workload prediction, a) The performance of HRV features decreases with epoch size, however, b) this drop in performance can be compensated by the fusion of breathing features with HRV