Facial expressions around the mouth and speech are essential elements in communication. However, due to mental and physical factors such as one’s personality, physical condition, and natural appearance, someone may not perform to their full potential. To solve this problem, we propose a mask-type device called “MouthOver” that substitutes for speech and facial expressions. This device is composed of a mask with a speaker and a display. It presents a prerecorded voice and a computer-processed mouth image, and allows users to realize context-specific speech and facial expressions regardless of their mental or physical state.

We also implemented “HappyMouth” that further extended human face-to-face communication capabilities. It has the functions of displaying one’s favorite mouth, converting the utterances into text and displaying it as subtitles, and displaying the results of Internet image searches for the keywords uttered. We evaluated the impressions of the wearers of the proposed device, and they were more attractive and positive, and gave a warmer impression than those who wore ordinary masks. It was also found that the wearers of the ordinary masks felt a more bland impression than those who wore the proposed system.

Publication

Ayaka Ishii, Takanori Komatsu, and Sunao Hashimoto. HappyMouth: Augmentation of Face-to-Face Communication Ability with a Mask-type Device, IPSJ SIG Technical Reports, No. 7, pp. 1-7, 2018 (in Japanese) [doi] [PDF]

Ayaka Ishii and Sunao Hashimoto. MouthOver: A Wearable Device to Substitute Voice and Mouth, 
 IPSJ Interaction, pp. 844-845, 2017(in Japanese) [PDF]