DATA:

Collection:

To assess the food decision-making processes in both environments, behavioral assessments and quantitative measurements were adopted, including questionnaires and various sensing modalities such as prefrontal cortex functional near-infrared spectroscopy (fNIRS), electrocardiography (ECG), galvanic skin response (GSR), eye movement and body motion. Neurophysiological measures were captured in both VR and RL phases of the study. Neurological signals from the prefrontal cortex of the brain during food selection were captured using an OCTAMON 8-channel fNIRS device. In the VR setting, participants’ physiological signals, including heart rate (ECG), GSR and position were measured using Shimmer wearable sensors. In both the VR and RL setting, Microsoft band was also used to capture skin temperature, heart rate (PPG), GSR, accelerometer and gyroscope data. Furthermore, the nutrition density of the food which participants chose in both environments was calculated and categorized into two groups (high and low-density). Then, data processing techniques were developed to identify the neurophysiological variations with different food choices (high/low density) in both environments. There are 15 Participants involved in VR and RW data collection process, which are collected one week apart.

Explanation and EDA:

There are certain tasks involved to prior and post the data collection process for better understanding of participants state of mind.

1. Hunger Emotion Rating - to know how hungry is the participant.

2. Baseline Task - to obtain the baseline of heart rate.

3. Go / no-Go Task - to practice and instruct the participants about VR setting and Data collection.

4. Speech Emotion Task - Surprise participants and ask to talk about their strength and weakness, which can help understand their physical and mental state.

5. VR and RW food selection task.

6. Review Questionnaire.

Comprising all of these together we have a TIMESTAMP DATA, recording the start and end times of above events, which helps to data instances from respective interval to analyze particular task.

fNIRS Data (prefrontal cortex functional near-infrared spectroscopy):

To assess the Neurological behavior during food selection in both VR and RW buffet environment, fNIRS sensors are used with two different receivers, 8 different light sources with two different wavelengths measuring the oxygenated, deoxygenated hemoglobin content in the prefrontal cortex of brain.

Most challenging part of this project is preprocessing fNIRS data to obtain a high quality signal by removing artefacts like,

1. Channel Exclusion - reason of this noise in signal is improper coupling of opt odes to head, instrument and surrounding noise.

2. Motion artefacts - this is the noise generated by head movements or the body movements.

3. Physiological artefacts - these are generated by Cerebral signals or Skin Hemodynamics.

We are yet to implement these preprocessing methodologies on this fNIRS data, exploring those tools and software's available.

After spend most my time in research for the optimal methods to apply to obtain a best quality signal we have decided to use Wavelet Filtering to avoid Motion artefacts and SQI(Signal Quality Indexing) to eliminate Channel exclusion and Physiological artefacts.

Shimmer's Data (Physiological Data):

To analyze the physiological behavior, they have used Shimmer's sensors and Microsoft band to detect the motion using Accelerometer, GSR (Galvanic Skin Response) to sense skin gland activity, ECG (Electrocardiography) to know the heart rate and Gyroscope to sense movement.

I have worked on two different types of Physiological data collected using shimmer device.

1. GSR Data (Galvanic Skin Response) measures the skin conductance of sweat gland activity

2. ECG Data (Electro Cardio Graph) helps to measure the heart rate variability

Our goal here is to understand Stress Indicating Factors (GSR&ECG) and discriminate each event period using the features extracted from GSR & ECG:

Below is the Image representing three different event periods namely VR Baseline, Speech Emotion and Food Selection on a sample GSR signal.

METHODOLOGY:

1. Understand Data Collection procedures and Data Formatting.

2. Data preprocessing to extract High Quality Signals.

3. Feature Extraction to create Metadata required to formulate ML models.

4. Automate pre-processing and Feature extraction for whole data (different subjects).

5. Formulate machine learning model like to discriminate event periods.

PHASE II:

Analyzing GSR (Galvanic Skin Response) and Electrocardiogram (ECG) data to compare how stress indicating factors and other physiological patterns change during each event period, and understand the discrimination between VR Baseline, Speech Emotion and VR Food Selection event periods.

GSR or Electrodermal Activity:

GSR is used to measure sweat gland activity related to emotional arousal, it refers to variation of electric conductance of the skin in response to skin secretion.

Two major components of GSR signal are:

Skin Conductance Response (SCR): Also called the Phasic component, it refers to the number of sweat glands that are activated. SCRs are faster in their appearance and takes longer to decline to baseline, which means that another SCR occurs shortly one after another resulting in an overall increase in GSR activity. This cumulative effect leads to the underestimation of SCR amplitude.

The Rapid rise and asymptomatic exponential decay are the main characteristics of this signal and the overlap issue is the main problem in the decomposition of these signals.

a. SCR occurs in response to a stimulus which is an event.

b. NS-SCR- non-specific SCR occurs without any specific reason.

Skin Conductance Level (SCL): Also called the Tonic component, is a slowly changing and continuous signal. Slow drifts in baseline and spontaneous fluctuations are the characters of this signal.

Below is the image discriminating raw GSR signal and its Phasic and Tonic components.

Feature-Based Learning (Time Series Analysis):

Though various research suggests that Multi-variate analysis using VAR and LSTM on time series data could give better predictions, but converting a sequential set of measures to feature vectors better captures global and dynamic properties of time series. Instead of deriving from the shapes of time series subsequences, understanding comes from time-series patterns, discriminating time series intervals that perfectly suit our classification problem.

Feature Extraction from Processed GSR:

Next, we have to proceed further to understand and discriminate three different events using stress indicators factors that is, to extract various statistical features from a processed signal. Few of the statistical features include Mean, SD, Peaks, Mean, Median, AUC, Slope, etcetera.

I have extracted features from the Phasic and Tonic components of GSR signals using a Sliding window technique with an overlap of 50 and a window length of 100. Images representing Number of features extracted for each subject(1st Column is the Subject ID).

PHASE III:

ECG or Electro Cardio Graph:

This is heart related data and HRV features extracted from these signals are helpful to understand the stress analysis during food selection process. Figure classifying QRS complex in a heart signal.

HRV is the Heart Rate Variability computed using the location of R peaks present in the ECG signal. After extracting R peaks locations in a GSR signal, I have used sliding window technique with window length of 60 and overlap of 30 to extract hrv features. I have used hrv_time() function from python neurokit.py module to extract HRV features from ECG signal where R peaks are located. https://neurokit2.readthedocs.io/en/latest/functions.html#module-neurokit2.hrv Image representing how Event period is effecting the HRV in individuals, you can clearly see increase in stress reduces HRV during speech emotion task.

Images representing Number of features extracted for each subject(1st column is Subject ID).

Modeling using Cross-Validation:

I have labeled each feature parameter with a respective event at that particular timestamp using the time stamp data and carried out binary classification on VR Baseline, Speech Emotion, and VR Food Selection events conducting three different experiments using K-Fold CV to discriminate among the events. Experiments namely,

a. VR Baseline vs Speech Emotion

b. Speech Emotion vs VR Food Selection

c. VR Baseline vs VR Food Selection

Prediction Analysis:

GSR Feature Predictions:

1- We have carried out predictions to understand which features are better predciting the event period. Tried training models on Phasic&Tonic, just Phasic and just Tonic. Random forest model with 13 fold cross validation trained on both Phasic&Tonic data have predicted better results when compared to just Phasic and just Tonic models.

2- Experiment VR Baseline vs Speech Emotion have better predicted when compared to other two experiments.

3- Also analysis on the individual subjects using Leave One Group Out helps to understand individual behavior of subjects which indeed performed poor when compared to K-Fold CV when all the subjects combined.

Figure representing the Event wise discrimination of algorithms predicting the AUC score:

HRV Feature Predictions:

1-Similar to GSR Data HRV also exhibited the same out come, K-fold cross validation with Random Forest on the VR Baseline vs Speech Emotion experiment have worked better than other algorithms and experiments.

2- Also the individuals predictions have been similar to GSR but we can observe little but of stability among all the subjects during experiment VR Baseline vs Speech Emotion. Figure representing the Event wise discrimination of algorithms predicting the AUC score:

Out Comes:

1. Understanding and knowing whether GSR and ECG data are useful to analyze stress.

2. Effect of different features.

3. Confirmations for further research.

Future Proceedings:

1. Go to Task is to merge GSR and HRV features together and analyze results.

2. Working on other psychological features and collecting more data.

REFERENCES:

1. docs.google.com/document/d/1QtUN_rPPZWlJUvuM67UhrIwsja1iuWzeKvjud3tBQW4/edit

2. docs.google.com/document/d/1d2FHhuutn1zJODxxhgkYzXX8xxlVRwXizBLx3MLAphI/edit

3. arxiv.org/ftp/arxiv/papers/2005/2005.01834.pdf

4. cvxopt.org/userguide/solvers.html#problems-with-nonlinear-objectives

5. biosppy.readthedocs.io/_/downloads/en/latest/pdf/

6. arxiv-vanity.com/papers/1709.08055/