November 2, 2023

Facial Micro-Expression Recognition Using a Hybrid Model based on Convolutional LSTM and Vision Transformer

In the billions of faces that are shaped by thousands of different cultures and ethnicities, one thing remains universal: the way emotions are expressed. To take the next step in human–machine interactions, a machine (e.g., a humanoid robot) must be able to clarify facial emotions. Allowing systems to recognize micro-expressions affords the machine a deeper dive into a person’s true feelings, which will take human emotion into account while making optimal decisions. For instance, these machines will be able to detect dangerous situations, alert caregivers to challenges, and provide appropriate responses. Micro-expressions are involuntary and transient facial expressions capable of revealing genuine emotions. We propose a new hybrid neural network (NN) model capable of micro-expression recognition in real-time applications. Several NN models are first compared in this study. Then, a hybrid NN model is created by combining a convolutional neural network (CNN), a recurrent neural network (RNN, e.g., long short-term memory (LSTM)), and a vision transformer. The CNN can extract spatial features (within a neighborhood of an image), whereas the LSTM can summarize temporal features. In addition, a transformer with an attention mechanism can capture sparse spatial relations residing in an image or between frames in a video clip. The inputs of the model are short facial videos, while the outputs are the micro-expressions recognized from the videos. The NN models are trained and tested with publicly available facial micro-expression datasets to recognize different micro-expressions (e.g., happiness, fear, anger, surprise, disgust, sadness). The proposed hybrid model performs the best; furthermore, score fusion can dramatically increase recognition performance.

About the speaker

Dr. Yufeng Zheng is an associate professor of data science at the University of Mississippi Medical Center. He received his Ph.D. in optical engineering and image processing in 1997 from Tianjin University, China. From 2001 to 2005, he served as a postdoctoral research associate at the University of Louisville, Kentucky. Dr. Zheng holds a utility patent in face recognition and has authored or coauthored three books, six book chapters, 27 articles in peer-reviewed journals, and 61 papers in conference proceedings. He serves as the principal investigator on several funded projects, including cybersecurity enhancement with keyboard dynamics, canopy coverage estimation with neural networks, multisensory image fusion and colorization, thermal face recognition, and multispectral face recognition. Dr. Zheng is a Cisco Certified Network Professional (CCNP) and holds senior memberships with IEEE & Signal Processing Society and SPIE. His research interests span various areas, including image processing and pattern recognition, neural networks and artificial intelligence, information fusion, biometrics (facial recognition), machine learning and computer vision, and computer-aided diagnosis.