Demo Video of our data collection for the working memory dataset. The sensors used in our study are shimmer, emotiv headset and gaze tracking application along with ground truth video.

The Sensors

Stimulated Navigation Task Environments

Study Design

We designed two studies for the project to accomplish our overarching goal:- 

(i) Study 1: PHYSIOLOGY-DRIVEN EPISODE EXTRACTION AND VERBAL CUEING - For each stimulated navigation task, the participants were cued verbally to look out for specific objects during the navigation task, starting from the same initial point. The task period was restricted to 1 minute to ensure that the participants largely remained or navigated around the same areas for control. After a 5-minute maintenance period, the participants took the PAAS scale and were asked to recall the number of each object type (cued at the beginning of the task and non-cued) that they encountered.    

(ii) Study 2: NAVIGATION RETRACING - We designed a within-subjects study to understand the use of attention-driven cueing in unknown environments in recalling navigation routes. In particular, we focus on visual cueing as prior works found that while spatial and verbal secondary tasks can impair one's navigation performance, visual secondary tasks do not impact navigation performance across participants with both high and low navigation ability. 

Each participant performed three navigation tasks (indoor, campus and mid-US-city) randomly to eliminate any ordering effects. The participants then carry out the navigation task for a duration of 5 minutes following which they self-rated the cognitive load on the PAAS scale. After a 5 minute rest period, the participants received one of the three treatments: (1) Unaided memory (no treatment), (2) visual aids with no cueing where the participant's recorded movement during the task was replayed in its entirety, and (3) visual cues extracted by memento framework (i.e., frames extracted as corresponding to high attentional focus activities with highlighted regions where the participants' gaze were fixed during the task). Using Play/Pause/Drag controls, the participants can sift through the video within 1 minute. For cases (2) and (3), the participants were then administered the NASA TLX scale for evaluating the cognitive load induced by using a memento instead of reviewing the entire video. 

After treatment, the participants were asked to retrace their route in the virtual environment exactly as they had done before, within the same 1-minute window, similar to a Route Map Recall Test (RMTC). We compute the accuracy with which they remembered their route as a measure of recall. On completion, we provided an open-ended questionnaire for the participants to note down significant landmarks/objects they noticed during the navigation task. Finally, we ask the participants to note down their confidence in the tool (cueing vs. no cueing) in aiding them in completing the task of re-tracing under the given time constraints. 

Data Collection Procedure

Participants wore Emotiv Insight headsets equipped with EEG and IMU sensors and Shimmer sensors for PPG and GSR data. Screen recordings and gaze fixations were captured during the navigation stimuli tasks using the freely available GazeRecorder application. GazeRecorder features a simple user interface (UI) for gaze calibration, requiring users to follow red points on the screen to synchronize facial and gaze data. Fixation heatmaps were generated to identify areas of interest during tasks.

To synchronize data from the Shimmer and Emotive Insight devices, we noted each task's start and end times. At the beginning of each session, participants were instructed to "NOD" and "WAVE" their dominant hand three times to log session timestamps. Additionally, an action camera (Akaso) recorded each session as ground truth, providing a visual reference for participants’ activities and gaze patterns.

Publications

1. Indrajeet Ghosh, Kasthuri Jayarajah, Nicholas Waytowich, Nirmalya Roy. "EEGAmp+: Investigating the Efficacy of Functional Connectivity for Detecting Events in Low-Resolution EEG", In Proceedings of the 21st EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous), Oslo, Norway, Nov 2024. 

2. Indrajeet Ghosh, Garvit Chugh, Kasthuri Jayarajah, Nirmalya Roy. "β-Decode: Attention-based Decoding Temporal Artifacts via Unsupervised β-Variational Autoencoder", In Proceedings of the ACM 8th Joint International Conference on Data Science & Management of Data (12th ACM IKDD CODS and 30th COMAD), IIT Jodhpur, India, Dec 2024.

3. Cahree Myrick, Indrajeet Ghosh, Kasthuri Jayarajah, Nirmalya Roy. CALM: Multimodal Cognitive Load Assessment Framework via Engineered and Explainable Features, in Proceedings of 5th IEEE International Workshop on Human-Centered Computational Sensing, co-located with the 23rd IEEE International Conference on Pervasive Computing and Communications(PerCom), Washington DC, USA, March 2025.

Contact Information: Indrajeet Ghosh (indrajeetghosh@umbc.edu)

PI: Dr. Nirmalya Roy

Collaborators: Dr. Kasthuri Jayarajah and Dr. Nicholas Waytowich

This work has been partially supported by U.S. Army Grant #W911NF2120076, NSF CAREER Award #1750936, ONR Grant #N00014-23-1-2119, NSF REU Site Grant #2050999, and NSF CNS EAGER Grant #2233879.