Research

Research topic: Brain-Computer Interfaces

Brain-Computer Interface (BCI) are devices that enable its users to interact with computers by mean of brain-activity only, this activity being generally measured by ElectroEncephaloGraphy (EEG). A typical example of a BCI would be a system in which a user can imagine movements of his left or right hand in order to move a cursor on a computer screen towards the left or right respectively. BCI systems are a very promising tool for severely paralysed people such as those suffering from late-stage Amyotrophic Lateral Sclerosis (ALS). Indeed, for those people, BCI can be the only mean of communication with the external world. Some BCI research groups have actually designed BCI prototypes with which disabled people could operate a text editor or a prosthesis. However, BCI can also be a promising interaction tool for healthy people, with several potential applications in the field of multimedia, virtual reality or video games among many other potential applications.

While they are very promising for numerous applications, BCI remain barely used outside laboratories, notably due to their lack of robustness and practicality. My long-term research objective is to designing practical BCI technologies that could be used outside the lab. To do so, I target 3 main research directions:

• Designing EEG signal processing and machine learning algorithms that can ensure robust EEG command decoding with minimal calibration times
• Understanding user training for BCI and designing appropriate training tasks and feedback to support such training
• Exploring passive BCI applications and in particular neuroergonomics for 3D user interfaces

Signal Processing and Classification techniques for EEG-based BCI

Although Brain-Computer Interfaces (BCI) have demonstrated their tremendous potential in numerous applications, they are still mostly prototypes, not used outside laboratories. This is mainly due to the following limitations:

1. Performances: the poor classification accuracies of BCI make them inconvenient to use or simply useless compared to available alternatives
2. Stability and robustness: the sensibility of ElectroEncephaloGraphic (EEG) signals to noise and their inherent non-stationarity make the already poor initial performances difficult to maintain over time
3. Calibration time: the need to tune current BCI to each user’s EEG signals makes their calibration times too long.

As part of our research on EEG-based BCI, we notably aim at addressing these limitations by designing robust EEG signal processing and classification tools with minimal calibration times, in order to design practical BCI systems, usable and useful outside laboratories. To do so we explore the design of alternative features and robust spatial filtering algorithms to make BCI more robust to noise and non-stationarities, as well as more accurate. We also explore artificial EEG data generation and user-to-user data transfer to reduce calibration times.

Selected publications:

• F. Yger, M. Bérar, F. Lotte, "Riemannian approaches in Brain-Computer Interfaces: a review", IEEE Transactions on Neural System and Rehabilitation Engineering, 2017 - pdf
• F. Lotte, “Signal processing approaches to minimize or suppress calibration time in oscillatory activity-based Brain-Computer Interfaces”, Proceedings of the IEEE, vol. 103, no. 6, pp. 871-890, 2015 – pdf – code
• N. Caramia, F. Lotte, S. Ramat, “Optimizing spatial filter pairs for EEG classification based on phase synchronization”, International Conference on Audio, Speech and Signal Processing (ICASSP’2014), pp. 2049-2053, 2014 – pdf
• N. Brodu, F. Lotte, A. Lécuyer, “Exploring Two Novel Features for EEG-based Brain-Computer Interfaces: Multifractal Cumulants and Predictive Complexity“, Neurocomputing, vol. 79, no. 1, pp. 87-94, 2012, – pdf – code
• F. Lotte, C.T. Guan, “Regularizing Common Spatial Patterns to Improve BCI Designs: Unified Theory and New Algorithms“, IEEE Transactions on Biomedical Engineering, vol. 58, no. 2, pp. 355-362, 2011 – pdf – code
• F. Lotte, A. Lécuyer, B. Arnaldi, “FuRIA: An inverse Solution based Feature Extraction Algorithm using Fuzzy Set Theory for Brain-Computer Interfaces”, IEEE Transactions on Signal Processing, vol. 57, no. 8, pp. 3253-3263, 2009 - pdf
• M. Zhong, F. Lotte, M. Girolami, A. Lécuyer, "Classifying EEG for Brain Computer Interfaces Using Gaussian Processes", Pattern Recognition Letters, vol. 29, no. 3, pp. 354-359, 2008 - pdf
• F. Lotte, A. Lécuyer, F. Lamarche, B. Arnaldi, "Studying the use of fuzzy inference systems for motor imagery classification", IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 15, no. 2, pp. 322-324, 2007 - pdf
• F. Lotte, M. Congedo, A. Lécuyer, F. Lamarche, B. Arnaldi, "A Review of Classification Algorithms for EEG-based Brain-Computer Interfaces", Journal of Neural Engineering, 4, R1-R13, 2007 - pdf

User Training Approaches for Brain-Computer Interfaces (BCI)

Poor BCI performances are partly due to imperfect EEG signal processing algorithms but also to the user, who may not be able to produce reliable EEG patterns. Indeed, BCI use is a skill, requiring the user to be properly trained to achieve BCI control. If he/she cannot perform the desired mental commands, no signal processing algorithm could identify them. Therefore, rather than improving EEG signal processing alone, an interesting research direction is to also guide users to learn BCI control mastery. Our current work aims at addressing this objective. We notably explore theoretical models and guidelines from psychology and cognitive sciences about human learning in order to improve BCI training protocols. Our studies notably illustrate the theoretical limitations of current standard BCI training approaches and the need for alternative approaches. We also perform some actual experiments to further illustrate some limitations of current BCI training protocols and try to understand and analyse them. We notably study which users’ profile (personality and cognitive profile) fail or succeed at learning BCI control. Finally, we explore new feedback types and new EEG visualization techniques in order to help users to learn BCI control skills more efficiently. These new feedback and visualizations notably aim at providing BCI users with more information about their EEG patterns, in order to identify more easily relevant BCI control strategies, as well as motivating and engaging them in the learning task.

Selected publications:

• C. Jeunet, E. Jahanpour, F. Lotte, "Why Standard Brain-Computer Interface (BCI) Training Protocols Should be Changed: An Experimental Study", Journal of Neural Engineering, vol. 13, no. 3, 036024, 2016 - pdf
• C. Jeunet, B. N’Kaoua, F. Lotte, “Advances in User-Training for Mental-Imagery Based BCI Control: Psychological and Cognitive Factors and their Neural Correlates”, Progress in Brain Research, UK: Elsevier, vol 228, pp. 3-35, 2016 - pdf
• C. Jeunet, B. N’Kaoua, S. Subramanian, M. Hachet, F. Lotte, “Predicting Mental Imagery-Based BCI Performance from Personality, Cognitive Profile and Neurophysiological Patterns”, PLOS ONE, vol. 10, no. 12, e0143962, 2015 - pdf
• J. Schumacher, C. Jeunet, F. Lotte, “Towards Explanatory Feedback for User Training in Brain-Computer Interfaces”, IEEE International Conference on Systems Man and Cybernetics (IEEE SMC), 2015 – pdf
• C. Jeunet, C. Vi, D. Spelmezan, B. N’Kaoua, F. Lotte, S. Subramanian, “Continuous Tactile Feedback for Motor-Imagery based Brain-Computer Interaction in a Multitasking Context”, Interact 2015 – pdf
• F. Lotte, C. Jeunet, “Towards Improved BCI based on Human Learning Principles”, 3rd International Winter Conference on Brain-Computer Interfaces, invited paper, pp. 37-40, 2015 – pdf
• J. Mercier-Ganady, F. Lotte, E. Loup-Escande, M. Marchal, A. Lécuyer, “The Mind-Mirror: See your Brain in Action in your head Using EEG and Augmented Reality”, IEEE Virtual Reality conference (VR 2014), pp. 33-38, 2014 – video – pdf
• L. Bonnet, F. Lotte, A. Lécuyer, “Two Brains, One Game: Design and Evaluation of a Multi-User BCI Video Game Based on Motor Imagery”, IEEE Transactions on Computational Intelligence and AI in Games (IEEE T-CIAIG), vol. 5, num. 2, pp. 185-198, 2013
• F. Lotte, F. Larrue, C. Mühl, “Flaws in current human training protocols for spontaneous Brain-Computer Interfaces: lessons learned from instructional design”, Frontiers in Human Neurosciences, vol 7., no. 568, 2013 – pdf

Physiological computing to assess and optimize 3D User Interfaces

Recently, physiological computing has been shown to be a promising companion to Human-Computer Interfaces (HCI) in general, and to 3D User Interfaces (3DUI) in particular, in several directions. Among them, in our group, we are first interested in using various physiological signals, and notably EEG signals, as a new tool to assess objectively the ergonomic quality of a given 3DUI, to identify where and when are the pros and cons of this interface, based on the user’s mental state during interaction. For instance, estimating the user’s mental workload during interaction can give insights about where and when the interface is cognitively difficult to use. This could be useful for 2D HCI in general, and even more for 3DUI. Indeed, in a 3DUI, the user perception of the 3D scene – part of which could potentially be measured in EEG, is essential. Moreover, the usual need for a mapping between the user inputs and the corresponding actions on 3D objects make 3DUI and interaction techniques more difficult to assess and to design. Beyond evaluation alone, physiological computing could also improve existing 3DUI by increasing the symbiosis between the user and the interface, e.g., for visualization and analysis of large amounts of (3D) data.

Selected publications:

• J. Frey, M. Hachet, F. Lotte, "EEG-based Neuroergonomics for 3D User Interfaces: opportunities and challenges", Le Travail Humain, 2017 - pdf
• J Frey, M Daniel, J Castet, M Hachet, F Lotte, "Framework for Electroencephalography-based Evaluation of User Experience", ACM SIGCHI Conference on Human Factors in Computing Systems (ACM CHI), pp. 2283-2294, 2016 - pdf
• J. Frey, A. Appriou, F. Lotte, M. Hachet, "Classifying EEG Signals during Stereoscopic Visualization to Estimate Visual Comfort", Computational Intelligence and Neuroscience, Article ID 793975, 2015 - pdf
• D. Wobrock, J. Frey, D. Graeff, J.-B. de la Rivière, J. Castet, F. Lotte, “Continuous Mental Effort Evaluation during 3D Object Manipulation Tasks based on Brain and Physiological Signals”, Interact 2015 – pdf
• C. Mühl, C. Jeunet, F. Lotte, “EEG-based Workload Estimation Across Affective Contexts”, Frontiers in Neurosciences section Neuroprosthetics, vol 8, no. 114, 2014 – pdf
• J. Frey, C. Mühl, F. Lotte, M. Hachet. “Review of the Use of Electroencephalography as an Evaluation Method for Human-Computer Interaction”, International Conference on Physiological Computing Systems (PhyCS 2014), pp. 214-223, 2014 – pdf

Older work:

BCI for Interacting with Virtual Reality Applications

As haptics did a few years ago, BCI have the potential to completely revolutionize the way people interact with applications such as Virtual Reality (VR) applications. However, a considerable research effort must be done in order to actually use BCI for practical VR applications. Indeed, current BCI systems provide the user with a limited number of commands, thus impeding natural and efficient interaction. Also, VR is a specific kind of feedback which may affect the user and as such his mental states. As such, what are the effects of this feedback must be also studied. My research in this topic is dedicated to the design of new interaction techniques, to the evaluation of such techniques and to the study of the user experience while using a BCI for interacting with a VR application.

Selected publications:

• F. Lotte, J. Faller, C. Guger, Y. Renard, G. Pfurtscheller, A. Lécuyer, R. Leeb, "Combining BCI with Virtual Reality: Towards New Applications and Improved BCI", Towards Practical Brain-Computer Interfaces, Allison, B. Z.; Dunne, S.; Leeb, R.; Del R. Millán, J. & Nijholt, A. (ed.), Springer Berlin Heidelberg, pp. 197-220, 2013 - link - draft pdf
• Y. Renard, F. Lotte, G. Gibert, M. Congedo, E. Maby, V. Delannoy, O. Bertrand, A. Lécuyer, “OpenViBE: An Open-Source Software Platform to Design, Test and Use Brain-Computer Interfaces in Real and Virtual Environments”, Presence: teleoperators and virtual environments, vol. 19, no. 1, pp. 35-53, 2010 - draft pdf (color) - final pdf (black & white) - code
• F. Lotte, A. Van Langhenhove, F. Lamarche, T. Ernest, Y. Renard, B. Arnaldi, A. Lécuyer, “Exploring Large Virtual Environments by Thoughts using a Brain-Computer Interface based on Motor Imagery and High-Level Commands”, Presence: teleoperators and virtual environments, vol. 19, no. 1, pp. 54-70, 2010 - draft pdf (color) - final pdf (black & white)
• F. Lotte, J. Fujisawa, H. Touyama, R. Ito, M. Hirose, A. Lécuyer, "Towards Ambulatory Brain-Computer Interfaces: A Pilot Study with P300 Signals", 5th Advances in Computer Entertainment Technology Conference (ACE), 2009 - pdf
• J.B. Sauvan, A. Lécuyer, F. Lotte, G. Casiez, "A Performance Model of Selection Techniques for P300-Based Brain-Computer Interfaces", ACM SIGCHI Conference on Human Factors in Computing Systems (ACM CHI), pp. 2205-2208, 2009 - pdf
• A. Lécuyer, F. Lotte, R. Reilly, R. Leeb, M. Hirose, M. Slater, “Brain-Computer Interfaces, Virtual Reality, and Videogames”, IEEE Computer, vol. 41, no. 10, pp 66-72, 2008 - pdf

For a complete list of my publications, you can go there.

For interesting papers about BCI in general, you can go there.