Acknowledgement:

"This material is based upon work supported by the National Science Foundation under Grant No: 1915065 and 1915083 "

Disclaimer:

"Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation."

Award Information:

Award Title: SCH: INT: Collaborative Research: Detection, Assessment and Rehabilitation of Stroke-Induced Visual Neglect Using Augmented Reality (AR) and Electroencephalography (EEG)

Award Numbers: 1915065 and 1915083

Award Duration : 4 years (Estimated)

Investigators:

• Murat Akcakaya, PI of Award # 1915083, University of Pittsburgh
• Sarah Ostadabbas, PI of Award # 1915065 , Northeastern University
• Elizabeth Skidmore, co-PI of Award # 1915083, University of Pittsburgh
• George Wittenberg, co-PI of Award # 1915083, University of Pittsburgh

Students:

• Jennifer Mak, Bioengineering, University of Pittsburgh
• Xiaofei Huang, Electrical and Computer Engineering, Northeastern University
• Deniz Kocanaogullari, Electrical and Computer Engineering, University of Pittsburgh

Project Goals:

This project will develop a novel engineering and scientific approach that will capitalize on the recent advances in electroencephalography (EEG)-based brain computer interfaces, augmented reality (AR) systems, machine learning and information theory. Such a system will enable development and usage of visually evoked EEG-based BCIs in more naturalistic environments potentially during activities of daily living (ADL). Our approach is clinically motivated from unilateral spatial neglect. Unilateral spatial neglect is a perceptual disorder that is one of the most common consequences of right-side brain damage after stroke, occurring in 29% of the 15 million people who sustain stroke worldwide. Patients with neglect demonstrate inattention to stimuli on the contralesional side, often missing food on one side of the plate, missing words on one side of the page, bumping into the left door jamb, getting confused by moving objects, and being fearful of walking in crowded places. The current gold standard for detecting and rehabilitating neglect lacks generalizability to dynamic tasks and contexts encountered during ADL. To overcome these issues, our objective is to develop a system that will record EEG signals as a user views randomly appearing and disappearing targets on a head-mounted AR display. The system will detect and map visually neglected extrapersonal space with high accuracy through continuous EEG-guided neglect detection. Such a detection will then be used to trigger haptic, auditory, and visual stimulation while engaged in ``real-world'' tasks conducted during rehabilitation for reducing neglect-related disabilities. To achieve our objective, we have assembled a multidisciplinary team with engineering/computer science, occupational therapy, neurology, and neurorehabilitation expertise.

Broader Impacts:

Individuals with neglect after stroke require significantly longer rehabilitation lengths of stay, more direct treatment time provided by therapists, and greater long-term disability compared to individuals without neglect after stroke, which also culminates in significantly higher healthcare costs. Current rehabilitation practice lacks vehicles for systematic fidelity for detecting neglect and stimulating attention in real-time during ``real-world'' tasks. Our proposed system addresses these shortcomings in current practice. We anticipate that the successful outcome of this project will also affect the current practice on BCIs enabling design and implementation of such systems in more naturalistic environments and providing more immersive experiences. For example, immersive experiences are specifically important for obtaining more authentic behavioral and neurological responses during rehabilitation studies including individuals with emotion regulation disorders. Therefore, we also expect that our findings will lead do expansion of the use of BCI in many rehabilitation paradigms that are centered around the intervention for other neurological disorders. Moreover, through this project we aim to inspire current and future generations of students to pursue STEM careers by providing a combination of rigorous training and a variety of hands-on experiences that demonstrate how interdisciplinary collaborative teams provide opportunities for impact on each contributing field.

Current Results:

Developing the EEG-based AR system: We chose HoloLens1 as the augmented reality headset to display the “Starry Night” scheme, selected g.USBamp amplifier to collect the EEG data, set the PC as the main interactive terminal and regarded the PC as control unit to call each external device. The Arduino serves as a bridge to assist the PC to transmit triggers to Hololens1 over Bluetooth and to amplifier over cable. In the developed system, the PC needs to control the operation of HoloLens and Amplifier. In order to more accurately locate the time segment in a very long EEG signal sequence, when the target appears in the HoloLens view, PC controls and sends triggers to HoloLens to present targets and to the EEG amplifier to mark the EEG sequence being received at the same time. However, in practice, HoloLens only provides two wireless connection ways with PC for communication: WIFI and Bluetooth, while the amplifier is connected to the PC by a usb cable. This results in noticeably asynchronous trigger transmission delays of the HoloLens and EEG amplifier. We tested both the WIFI and Bluetooth connection options and identified that Bluetooth is much faster and much more reliable. There is still a small time difference between the amplifier and the AR headset. To solve this issue, we implemented a time correction algorithm.

Statistical test on EEG features: We analyzed the EEG data collected from 11 participants (5 stroke patients with neglect and 6 stroke patients without neglect). Through this analysis we were interested in identifying the specific EEG bands and/or electrodes that would most easily identify the response type (EEG corresponding to neglected/slow vs EEG corresponding to non-neglected/fast responses vs baseline EEG). First, topographical head plots of the energies of each band were created for the fast and slow responses and baselines of 1) individual participants, 2) neglect and no neglect groups, 3) all samples combined, regardless of subgroup. In the first set, the average energy of a particular band was plotted at each electrode for that particular response. In the second set, the same was done with all the participants averaged within each subgroup. In the third set, all the fast, slow, and baseline energies were averaged for all participants. Through a visual inspection, the greatest changes from baseline were found in the three frequency bands (delta, alpha, beta) and in 4-7 frontal electrodes and the occipital electrode. Our previous work used Wilcoxon signed-rank tests to evaluate the statistical differences in alpha, beta, and gamma band energies in left vs right hidden targets in healthy and healthy with simulated neglect participants. However, a similar statistical analysis was not done on the data collected on stroke participants with neglect. The dataset is further complicated by the fact that each participant has a different number of sessions, fast responses, and slow responses. Treating each target of each subject as a trial results in over 738,000 data points and creates many repeated measures per factor. However, this many trials increases the computational power required of programs and computers. Further statistical analysis must be conducted in order to find the most significant features when comparing the changes in baseline in the fast and slow responses in stroke participants with and without neglect. A generalized estimating equation (GEE), a regression approach, will be used as it is the best option for dependent trials, unbalanced repeated measures within groups or participants, and is flexible with non-normally distributed data. This analysis will be performed in Python and identify the electrodes and bands that result in the highest level of significance in fast and slow responses from baseline. Other traditional statistical softwares like SPSS and R were found to be unable to handle the computational power. A preliminary GEE analysis of main effects found three electrodes that had changes from baseline that were significantly different when comparing fast and slow responses: F3, F4 and Fp1. F3 and F4 were also identified to show large changes in baseline in the topographic plots, but not Fp1. Further analysis is needed to examine the interaction effects of the bands and electrodes.

Classification of neglected vs non-neglected targets using recorder EEG: The current gold standard for spatial neglect assessment is the behavioral inattention test (BIT). BIT includes a series of pen-and-paper tests. These tests can be unreliable due to high variability in subtest performances; they are limited in their ability to measure the extent of neglect, and they do not assess the patients in a realistic and dynamic environment. The proposed Starry Night test provides information about severity of spatial neglect. Moreover, it uses dynamic objects and distractors common in a real-life environment while the traditional tests are static. In the proposed test, electroencephalography (EEG) signals are recorded during presentation of visual stimuli in random locations on a screen with a background of dynamically changing distractors. By analyzing temporal features of EEG data collected from 5 patients with SN and 6 patients without SN, it was found that sensitivity of identifying SN is 88.89% with specificity and accuracy of 80.95% and 83.33% respectively. Moreover, experimental results showed that accuracy and specificity of the proposed test are correlated with BIT scores (R²=0.81 and 0.83 respectively) of the SN group. Moreover, it was found that the field of view estimated from EEG of the SN group is highly correlated with BIT scores (R² =0.92) [5], see the Figure below for the best and the worst case of field of view (FOV) estimation. Finally, we applied Convolutional Neural Network based deep learning model to identify EEG features not only in time domain but also in the spatial domain to improve the detection of neglect across stroke patients with and without neglect. Our result showed that such a detection can be achieved with an average accuracy 89.73%, average specificity 89.34% and average sensitivity 86.97% [3], [4].

Recursive Bayesian State Estimation: We have developed a recursive Bayesian state estimation method that enables stimulation optimization and feature selection for unknown state estimation. This method will be used in the second, third and fourth years of the project to optimize the location of the visual stimulations. Our work related to the recursive Bayesian state estimation were published in two different venues: (i) in the BCI Journal [1] and in the Proceedings of International Conference on Pervasive Technologies Related to Assistive Environments [2].

Awards, Prizes, Highlights and Press Release:

• News @University of Pittsburgh (October 23, 2019): Bringing Attention to Visual Neglect in Stroke
• News @Northeastern University (July 31, 2019): NSF Grant to combined Augmented Reality (AR) and EEG for Stroke-Induced Visual Neglect Rehabilitation

Publications:

1. A. Kocanaogullari, M. Yarghi, M. Akcakaya, and D. Erdogmus, “ An Active Recursive State Estimation Framework for Brain-Interfaced Typing Systems,” Brain Computer Interfaces, March 2020 (https://doi.org/10.1080/2326263X.2020.1729652)
2. A. Kocanaogullari, M. Akcakaya, B. Oken, and D. Erdogmus, “Optimal Modality Selection Using Information Transfer Rate for Event Related Potential Driven Brain Computer Interfaces,” PETRA 2020 (DOI: https://doi.org/10.1145/3389189.3389197)
3. D. Kocanaogullari, J. Mak, J. Kersey, A. Khalaf, S. Ostadabbas, G. Wittenberg, E. Skidmore, and M. Akcakaya,” EEG-based Neglect Detection for Stroke Patients,” IEEE EMBC 2020 (accepted for publication, presented virtually but not available online yet.)
4. M.Akcakaya, D. Kocanaogullari, A. Khalaf, J. Kersey, J. Mak, X. Huang, S. Ostadabbas, G. Wittenberg, and E. Skidmore, “ An EEG-based BCI for Visual Spatial Neglect Detection and Assessment,” BCI Meeting 2020 (Abstract accepted but the conference/workshop postponed due to COVID-19).
5. A.Khalaf, J. Kersey, S. Eldeeb, G. Alankus, E. Grattan, L. Waterstram, E. Skidmore, and M. Akcakaya, “ A Passive EEG-based Brain Computer Interface for Assessment of Visuospatial Neglect,” BCI Journal (under review)

Presentations:

1. A Passive EEG-based Brain Computer Interface for Assessment of Visuospatial Neglect, Virtually presented at Minisymposium on Artificial Intelligence in Rehabilitation at IEEE EMBC 2020
2. A Passive EEG-based Brain Computer Interface for Assessment of Visuospatial Neglect, was accepted to be presented at the Workshop on BCI in Stroke Rehabilitation at the BCI Meeting 2020 (the conference is postponed due to COVID-19)

Data, Software and Demos:

(TBD)

Educational Material:

(TBD)

Point of Contact: Murat Akcakaya: akcakaya[at]pitt[dot]edu

Last updated: 07/15/2020