BioPhyS

Media and Publications

Muñoz, J. E., Quintero, L., Stephens, C. L., & Pope, A. T. (2020). A Psychophysiological Model of Firearms Training in Police Officers: A Virtual Reality Experiment for Biocybernetic Adaptation. Frontiers in Psychology, 11. (Link)

Muñoz, J. E., Pope, A. T., & Velez, L. E. (2016). Integrating biocybernetic adaptation in virtual reality training concentration and calmness in target shooting. In Physiological Computing Systems (pp. 218-237). Springer, Cham. (Link)

System Description

Based on NASA Langley patents (Alan Pope and colleagues) licensed by the J&F Alliance Group company as part of the NASA Technology Transfer Program, I was leading the design and development process of an intelligent system that uses sophisticated biofeedback technologies to enhance cognitive skills in virtual military training. The system integrates novel Virtual Reality (VR) and Biocybernetic technologies to deliver highly adaptive scenarios aiming at boosting the training of important cognitive skills in the military personnel. 

Design Process

A Wizard of Oz (WOz) panel was implemented allowing trainers to set up different scenarios by means of modifying a set of predefined simulation variables. The targets can be modified in its size, hardness, speed and quantity while environmental variables such as rain intensity and daylight can be also modified. This panel was also useful to integrate the physiological modulation layer by means of the biocybernetic loop engine. 

Modulation Experiment and Biocybernetic Loop Engine Integration

The Biocybernetic Loop is the system adaptation layer that will be in charge of using physiological signals to create real-time and automatic adaptations into the simulator based on the detection of concentration and calmness levels of users. First, we decided to use two types of physiological signals: neurophysiological and cardiovascular. Both approaches will allow us to individually quantify concentration and calmness levels of users in order to modulate the simulation difficulty to promote self-regulation skills (Biocybernetic Adaptation). We noticed that an important factor that needs many hours of training is the breathing pace. The system should encourage users to carry out respiration techniques that will induce optimal levels of concentration and calmness, which are optimal to reduce avoidable mistakes while shooting. Training a natural breathing pattern (autogenic breathing) aiming at keeping optimal levels of oxygen in the blood has been defined as a major component of firearms training. We used the Muse BCI wearable sensor and the Polar H10 chest strap to record both neurophysiological and cardiovascular signals. 

In order to add this intelligence to the system, we rely on the Biocybernetic Loop Engine, a software tool I designed years ago in Unity3D that aids the integration of physiological adaptation to games and simulations. To integrate the biocybernetic adaptation technology to the BioPhyS simulation, we have to include a Unity prefab into the Unity project which contains two scripts that allow a bidirectional communication between the Biocybernetic Loop Engine and the simulation (manual here). The software currently supports cardio (Polar, Android Smartwatches and Plux devices), facial expressions (affectiva SDK) and wearable BCI sensors (Mindwave, Muse BCI, Emotiv Epoc). 

Before including the biocybernetic loop, we carried out a physiological characterization study with 10 young male users to measure their cardiovascular and neurophysiological responses during different configurations of the target-shooting simulator as well as to get data from their perceived difficulty and motion sickness after interacting with the system for about 30 minutes. The following presentation summarizes the main findings of the experiment.

Experimental Psychophysiological Model

After having real physiological data from users during a short interaction with the BioPhyS simulator, we create a psychophysiological model of the system that allowed us to create the biocybernetic adaptation system. In the experiment, we found that the frontal alpha bandpower was significantly different between the baseline and the active shooting conditions simulated in the experiment. We associated this biomarker with the concentration levels during the target-shooting (more frontal alpha activity = more concentration). Similarly, we found that heart beats (and other HRV parameters) were significantly affected by the simulation difficulty, so we associate HR to calmness levels. A simplified psychophysiological model was created for this. 

After having a simplified and concise model of the psychophysiological behavior of users during the target-shooting simulation, we defined the variables that would be modulated by means of the biocybernetic loop system. Beyond the target-based variables (size, hardness, number, horizontal speed), we added two environmental variables that would be triggered by means of users' calmness and concentration levels, rain intensity and daylight. After having the model and the communication with the Biocybernetic Loop Engine, we can create adaptation rules based on a simplistic if/then logic for triggering the simulation variables in the BioPhyS target-shooting simulator. 

To summarize, after integrating the Biocybernetic Loop Engine with the BioPhyS VR simulator for target-shooting, we are able to create physiologically adaptive training scenarios aiming at persuading trainees to master self-regulation skills that promote good concentration and calmness levels (e.g. autogenic respiration) while shooting in VR. The system is able to modify the simulation variable in real-time to produce highly physiologically adaptive scenarios. For instance, one would like to create metaphors such as increasing the rain intensity once you are getting dangerously excited during the target-shooting, or increasing the simulation difficulty by means of changing from static targets to moving targets once the user reaches good levels of concentration. 

Explicit biofeedback and Post Session Analysis

Two final features were added to the system based on an intense iterative design process. Firstly, since the physiological modulation is not happening instantaneously (remember we use windows of X seconds to average the data and  trigger the events), sometimes users were unable to control their breathing/concentration properly simply because they did not have real-time biofeedback. To solve this, we added an extra block in the Biocybernetic Loop Engine able to stream the raw physiological data and place it into the simulation by using dynamic bar plots. Those bars represented the Concentration and Calmness levels and are always visible to the users in order to better adjust their self-regulation strategies. Finally, the streamed data is store and the system automatically produce plots that can be used for a post-analysis session of both, physiological and simulation variables (e.g. effectiveness, times, performance).

Air Pistol Integration (Vive Tracker)

A picatinny-like support was build to integrate weapons to the BioPhyS simulator in order to provide haptic feedback and more realism. The picatinny rail mount used a Vive Tracker, a mounting bracket and a mounting bolt holder to integrate pistols to the VR simulation. We are initially using the Sauer P320 CO2 Air Pistol to adjust the picatinny rail with the Vive tracker, but similar weapons can be also used. For the trigger, a Hall effect sensor was coupled to the real weapon's trigger, sending the shooting signal to a micro controller. An initial approach uses the voltage from the Vive tracker to supply power to the micro controller that receives and processes the information. 

Pilot Study with Police Officers

In collaboration with the Hampton Police Department, we are currently carrying out training sessions with police officers to measure their responses and create the psychophysiological model for this population. After having the model, we will test different versions of the biocybernetic loop and we will evaluate their effectiveness in maximizing concentration and calmness during target-shooting. For instance, we can use the 100-115 bpm targeted zone to quantify what is the best physiologically modulated system that can maximize the time police officers spend in that desirable zone while shooting in VR. This might produce a considerable effect in the officers' performance by means of mastering their sympathetic (fight and flight) responses, which are responsible to keep them alert without loosing important fine motor and visual skills. 

People Involved

After my 10-weeks long Internship in Hampton Virginia, I had the opportunity to interact with several very skilled people that helped me to design the more robust biocybernetic system that I ever have done in my entire career. All my research was always supervised by my academic hero, Dr. Alan Pope (now retired from NASA Langley) who was always supportive and incrementally "positive" along the whole project. His ideas were a key supply for integrating the sophisticated biocybernetic loop technology he just pioneered back in the 80-90's. His PRISM team at NASA was also a very trustable source of feedeback and I'm looking forward to keep my collaboration with them.  The very sharp Unity developer and friend from Colombia, Luis Quintero was in charge of the development process. Without his very efficient way to work and accelerated learning process, I hadn't been able to deliver such functional, robust and demo-able technology. All in all, he was one of the cornerstones in the development of the Biocybernetic Loop Engine back in Portugal on 2017. Finally in the host institution (J&F Alliance Group company) I found a very respectful and encouraging environment to move my technology out-of-the-lab, understanding the very different times and pressures in a company once compared with a research lab. The lovely couple of JarMarcus and Falana King (COO and CEO) open their company (and their wallet certainly) to allow my visit as an International Visitor Study through the National Institute of Aerospace. Jeremy Sklute who was in charge of integrating the air-pistol and with whom I established a very "hostile" ongoing war with his Nerf-Guns arsenal inside the company. Michael Priddy ("Captain America") who adopted me on his home with his very adorable and welcoming family. With Michael I had the opportunity to learn the very basics of firearms and shooting and I got a very intense experience in a real target-shooting indoor range (I was lucky since "Mike" has years and years of military training as US Veteran).