Bio

He received his M.S. degree in engineering from Tokyo University of Agriculture and Technology and joined NTT Corporation in 2008. He received his Ph.D. degree in informatics from Kyoto University in 2013. He is currently a senior research scientist at NTT Media Intelligence Laboratories and a visiting scholar at Carnegie Mellon University.

Research Topics

 Multimodal Interaction, Multimodal Machine Learning, Conversation Analysis, Social Signal Processing

Awards (selected)

1. Collaborate Award, NTT Media Intelligence Research Laboratories, NTT Corporation, 2019

2. Press Award, NTT Media Intelligence Research Laboratories, NTT Corporation, 2019

3. Feature Theme Session Award, HCG Symposium 2018, Institute of Electronics, Information and Communication Engineers (IEICE), 2018

4. President's Special Award, NTT Service Innovation Research Laboratories, NTT Corporation, 2018

5. Collaborate Award, NTT Media Intelligence Research Laboratories, NTT Corporation, 2018

6. Noguchi Award (1st Prize), Multimedia, Distributed, Cooperative, and Mobile Symposium, Information Processing Society of Japan, 2018

7. Outstanding Interactive Presentation Award, HCG Symposium, Institute of Electronics, Information and Communication Engineers (IEICE), 2015

8. Outstanding Interactive Presentation Award, HCG Symposium, Institute of Electronics, Information and Communication Engineers (IEICE), 2015

9. Feature Theme Session Award, HCG Symposium 2015, Institute of Electronics, Information and Communication Engineers (IEICE), 2015

10. Human Communication Award (HC Award), Human Communication Group (HCG), Institute of Electronics, Information and Communication Engineers (IEICE), 2014

11. Outstanding Paper Award, The 16th ACM International Conference on Multimodal Interaction (ICMI), 2014

12. FIT Encouragement Award, The 13th Forum on Information Technology (FIT), 2014

13. R&D Incentive Award, NTT Cyber Space Laboratory, NTT Corporation, 2011

14. Outstanding Demo Award, NTT R&D Forum 2011, NTT Corporation, 2011

Publications

Journal articles

1. Ryo Ishii, Ryuichiro Higashinaka, Koh Mitsuda, Taichi Katayama, Masahiro Mizukami, Junji Tomita, Hidetoshi Kawabata, Emi Yamaguchi, Noritake Adachi, Yushi Aono, "Methods of Efficiently Constructing Text-dialogue-agent System using Existing Anime Character",  Journal of Information Processing , 2021. (accepted) 

2. Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Ryuichiro Higashinaka, Yushi Aono, "Estimation of Personal Empathy Skill Level using Dialogue Act and Eye-gaze during Turn-keeping/changing", Journal of Information Processing , 2021. (in Japanese) (accepted) 

3. Ryo Ishii, Kazuhiro Otsuka, Ryuichiro Higashinaka, Junji Tomita, "Prediction of Who Will Be Next Speaker and When Using Mouth-Opening Pattern in Multi-Party Conversation",  Multimodal Technologies Interact, 3(4), 70, 2019.

4. Takahiro Matsumoto, Mitsuhiro Goto, Ryo Ishii, Tomoki Watanabe, Tomohiro Yamada, Michita Imai, “Spatial Arrangement Effects to Participant Workload in Human Multi-robots Conversation”, Journal of Information Processing of Japan, Vol. 60 No. 2, pp. 1–14, 2019. (in Japanese)

5. Ryo Ishii, Shiro Kumano, Kazuhiro Otsuka, “Estimating Empathy Skill Level using Gaze Behavior depending Participant-roll during Turn-changing/keeping”, The Transactions of Human Interface Society, No. 20, Vol. 4, pp. 447–456, 2018. (in Japanese)

6. Shiro Kumano, Kazuhiro Otsuka, Ryo Ishii, and Junji Yamato, “Collective First-Person Vision for Automatic Gaze Analysis in Multiparty Conversations”, IEEE Transactions on Multimedia, Vol. 19, No. 1, pp. 107–122, 2017.

7. Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Junji Yamato, "Using Respiration of Who Will be the Next Speaker and When in Multiparty Meetings", The ACM Transactions on Interactive Intelligent Systems (TiiS), Article No. 20, 2016.

8. Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Junji Yamato, "Prediction of Who Will be Next Speaker and When using Gaze Behavior in Multi-Party Meetings", The ACM Transactions on Interactive Intelligent Systems (TiiS), Vol. 6 Issue 1, Article No. 4, 2016.

9. Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Junji Yamato, “Predicting Next Speaker Based on Head Movement in Multi-party Meetings”,  Journal of Information Processing of Japan, Vol. 56, No. 4, pp. 1116–1127, 2016. (in Japanese)

10. Ryo Ishii, Shiro Ozawa, Akira Kojima, Kazuhiro Otsuka, Yuki Hayashi, Yukiko, I. Nakano, “Design and Evaluation of Mirror Interface MIOSS to Overlay Remote 3D Spaces”, The Transactions of Human Interface Society, Vol. 17, No. 3, 2015. (in Japanese)

11. Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Masafumi Matsuda, Junji Yamato, “Predicting Next Speaker and Timing from Gaze Transition Patterns in Multi-Party Meetings”, The Transactions of the Institute of Electronics, Information and Communication Engineers (IEICE), Vol. J97-A, No. 6, pp. 453–468, 2014. (in Japanese)

12. Ryo Ishii, Shiro Ozawa, Harumi Kawamura, Akira Kojima, Yukiko, I. Nakano, “Effectiveness of Remote Instructor's Action through a Window Interface MoPaCo”,  The Transactions of the Institute of Electronics, Information and Communication Engineers (IEICE), Vol. J96-D, No. 12, pp. 3044–3054, Dec. 2013. (in Japanese)

13. Ryo Ishii, Yukiko, I. Nakano, Toyoaki Nishida, “Gaze Awareness in Conversational Agents: Estimating User's Conversational Engagement using Eye-gaze”, The ACM Transactions on Interactive Intelligent Systems (TiiS), Special issue on interaction with smart objects, Special section on eye gaze and conversation archive, Vol. 3 Issue 2, 2013.

14. Ryo Ishii, Shiro Ozawa, Harumi Kawamura, Akira Kojima, Yukiko, I. Nakano, “MoPaCo : Window Interface to Enhance Telepresence in Video Communication”, IEICE Transactions on Information and Systems, Vol. J96-D, No. 1, pp. 110–119, 2013. (in Japanese)

15. Ryo Ishii, Ryoko Ooko, Yukiko, I. Nakano, Toyoaki Nishida, “Estimation of User’s Conversational Engagement Using Eye-gaze and Head Movement Information”, Journal of Information Processing of Japan, Vol. 52, No. 12, pp. 3625–3636, 2011. (in Japanese)

16. Ryo Ishii, Yukiko, I. Nakano, “Estimating the Degree of Conversational Engagement Based on User's Gaze Behaviors in Human-agent Interactions: Towards Adaptive Dialogue Management in Conversational Agents”,  Journal of Information Processing of Japan, Vol. 49, No. 1, pp. 3835–3846, 2009. (in Japanese)

17. Ryo Ishii, Toshimitsu Miyajima, Kinya Fujita, “Avatar’s Gaze Control to Facilitate Conversation in Virtual-Space Multi-User Voice Chat System”,  The Transactions of Human Interface Society, 10, 1, 2008. (in Japanese)

Refereed international conference papers

1. Chaitanya Ahuja, Dong Won Lee, Ryo Ishii, Louis-Philippe Morency. "No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform  Gestures ", Finding at EMNLP, 2020. (accepted)

2. Toshiki Onishi, Arisa Yamauchi, Ryo Ishii, Yushi Aono, Akihiro Miyata, "Analyzing Nonverbal Behaviors along with Praising",  Proceedings of ACM International Conference on Multimodal Interaction (ICMI), pp.609-613, 2020. 

3. Ryo Ishii, Chaitanya Ahuja, Yukiko Nakano, Louis-Philippe Morency, "Impact of Personality on Nonverbal Behavior Generation", Proceedings of ACM International Conference on Intelligent Virtual Agents (IVA), No. 29, pp.1-8, 2020.

4. Ryo Ishii*, Xutong Ren*, Michal Muszynski, Louis-Philippe Morency, "Can Prediction of Turn-management Willingness Improve Turn-changing Modeling?", Proceedings of ACM International Conference on Intelligent Virtual Agents (IVA), No. 28, pp.1-8, 2020. (* denotes joint first-authors) 

5. Ryo Ishii, Ryuichiro Higashinaka, Koh Mitsuda, Taichi Katayama, Masahiro Mizukami, Junji Tomita, Hidetoshi Kawabata, Emi Yamaguchi, Noritake Adachi, Yushi Aono, ”Methods of Efficiently Constructing Text-Dialogue-Agent System Using Existing Anime Character”, Proceedings of International Conference on Human-Computer Interaction (HCII), pp. 328-347, 2020. 

6. Ryo Masumura, Mana Ihori, Tomohiro Tanaka, Atsushi Ando, Ryo Ishii, Takanobu Oba, Ryuichiro Higashinaka, "Improving speech-based end-of-turn detection via cross-modal representation learning with punctuated text data", Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1062-1069, 2019.

7. Fumio Nihei, Yukiko Nakano, Ryuichiro Higashinaka, Ryo Ishii, "Determining Iconic Gesture Forms based on Entity Image Representation",  Proceedings of ACM International Conference on Multimodal Interaction (ICMI), pp. 419-425, 2019.

8. Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Ryuichiro Higashinaka, Junji Tomita, "Estimating Interpersonal Reactivity Scores Using Gaze Behavior and Dialogue Act During Turn-Changing", Proceedings of International Conference on Human-Computer Interaction (HCII), pp. 45-53, 2019

9. Ryo Ishii, Taichi Katayama, Ryuichiro Higashinaka, Junji Tomita, “Automatic Generation System of Virtual Agent's Motion using Natural Language”, Proceedings of ACM International Conference on Intelligent Virtual Agent (IVA), pp. 357–358, 2018.

10. Ryo Ishii, Taichi Katayama, Ryuichiro Higashinaka, Junji Tomita, “Generating Body Motions using Spoken Language in Dialogue”, Proceedings of ACM International Conference on Intelligent Virtual Agents (IVA), pp. 87–92, 2018.

11. Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Ryuichiro Higashinaka, Junji Tomita, “Analyzing Gaze Behavior and Dialogue Act during Turn-taking for Estimating Empathy Skill Level”, Proceedings of ACM International Conference on Multimodal Interaction (ICMI), pp. 31–39, 2018.

12. Ryo Ishii, Taichi Katayama, Ryuichiro Higashinaka, Junji Tomita, “Automatic Generation of Head Nods using Utterance Texts”, Proceedings of IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 1143–1149, 2018.

13. Ryo Ishii, Ryuichiro Higashinaka, Kyosuke Nishida, Taichi Katayama, Nozomi Kobayashi, Junji Tomita, “Automatically Generating Head Nods with Linguistic Information”, Proceedings of HCI International (International Conference on Social Computing and Social Media), pp. 383–391, 2018.

14. Takahiro Matsumoto, Mitsuhiro Goto, Ryo Ishii, Tomoki Watanabe, Tomohiro Yamada, Michita Imai, “Where Should Robots Talk?: Spatial Arrangement Study from a Participant Workload Perspective”, Proceedings of ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 270–278, 2018.

15. Ryo Ishii, Ryuichiro Higashinaka, Junji Tomita, “Predicting Nods by using Dialogue Acts in Dialogue”, Proceedings of Eleventh International Conference on Language Resources and Evaluation (LREC), pp. 2940-2944, 2018.

16. Ryo Masumura, Tomohiro Tanaka, Atsushi Ando, Ryo Ishii, Ryuichiro Higashinaka, Yushi Aono, “Neural Dialogue Context Online End-of-Turn Detection”, Proceedings of Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL), pp. 224–228, 2018.

17. Ryo Ishii, Shiro Kumano, Kazuhiro Otsuka, “Analyzing gaze behavior during turn-taking for estimating empathy skill level”, Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI), pp. 365–373, 2017.

18. Shiro Kumano, Ryo Ishii, Kazuhiro Otsuka, “Comparing empathy perceived by interlocutors in multiparty conversation and external observers”, Proceedings of Affective Computing and Intelligent Interaction (ACII), pp. 50–57, 2017.

19. Shiro Kumano, Ryo Ishii, Kazuhiro Otsuka, “Computational model of idiosyncratic perception of others”, Proceedings of Affective Computing and Intelligent Interaction (ACII), pp. 42–49, 2017.

20. Ryo Ishii, Shiro Kumano, Kazuhiro Otsuka, “Prediction of Next-Utterance Timing using Head Movement in Multi-Party Meetings”, Proceedings of the 5th International Conference on Human Agent Interaction (HAI), pp. 181–187, 2017.

21. Ryo Masumura, Taichi Asami, Hirokazu Masataki, Ryo Ishii, Ryuichiro Higashinaka, “Online end-of-turn detection from speech based on stacked time-asynchronous sequential networks”, Proceedings of INTERSPEECH, pp. 1661–1665, 2017.

22. Ryo Ishii, Shiro Kumano, Kazuhiro Otsuka, “Analyzing mouth-opening transition pattern for predicting next speaker in multi-party meetings”, Proceedings of the 18th ACM International Conference on Multimodal Interaction (ICMI), pp. 209–216, 2016.

23. Ryo Ishii, Shiro Kumano, Kazuhiro Otsuka, "Multimodal Fusion using Respiration and Gaze for Predicting Next Speaker Based in Multi-Party Meetings", International Conference on Multimodal Interaction (ICMI), pp. 99–106, 2015.

24. Ryo Ishii, Shiro Ozawa, Akira Kojima, Kazuhiro Otsuka, Yuki Hayashi, Yukiko I. Nakano, "Design and Evaluation of Mirror Interface MIOSS to Overlay Remote 3D Spaces", IFIP International Conference on Human-Computer Interaction (INTERACT), pp. 319–326, 2015.

25. Shiro Kumano, Kazuhiro Otsuka, Ryo Ishii, and Junji Yamato, "Automatic Gaze Analysis in Multiparty Conversations based on Collective First-Person Vision", International Workshop on Emotion Representation, Analysis and Synthesis in Continuous Time and Space (EmoSPACE), Vol. 5, pp. 1–8, 2015.

26. Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Junji Yamato, "Predicting Next Speaker Based on Head Movement in Multi-Party Meetings", International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2319–2323, 2015.

27. Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Junji Yamato, "Analysis of Timing Structure of Eye Contact in Turn-changing ", Workshop on Eye Gaze in Intelligent Human Machine Interaction: Eye-Gaze and Multimodality (Gaze-In), pp. 15–20, 2014.

28. Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Junji Yamato, "Analysis of Respiration for Prediction of "Who Will Be Next Speaker and When?" in Multi-Party Meetings", International Conference on Multimodal Interaction (ICMI), pp. 18–25, 2014. [Outstanding Paper Awards]

29. Ryo Ishii, Shiro Ozawa, Harumi Kawamura, Akira Kojima, Yukiko I. Nakano, Kazuhiro Otsuka, "Evaluation of Window Interface in Remote Collaboration involving Pointing Gesture", International Conference on Advances in Computer-Human Interactions (ACHI), 2014.

30. Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Junji Yamato, "Analysis and Modeling of Next Speaking Start Timing in Multi-party Meetings", International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 694–698, 2014.

31. Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Masafumi Matsuda, Junji Yamato, "Predicting Next Speaker and Timing from Gaze Transition Patterns in Multi-Party Meetings", International Conference on Multimodal Interaction (ICMI), pp. 79–86, 2013.

32. Kazuhiro Otsuka, Shiro Kumano, Ryo Ishii, Maja Zbogar, Junji Yamato, "MM+Space: n x 4 Degree-of-Freedom Kinetic Display for Recreating Multiparty Conversation Spaces", International Conference on Multimodal Interaction (ICMI), pp. 389–396, 2013.

33. Shiro Kumano, Kazuhiro Otsuka, Masafumi Matsuda, Ryo Ishii, Junji Yamato, "Using A Probabilistic Topic Model to Link Observers' Perception Tendency to Personality", Proceedings of HUMAINE Assoc. Conf. on Affective Computing and Intelligent Interaction (ACII), 2013.

34. Ryota Ooko, Ryo Ishii, Yukiko I. Nakano, “Estimating a User’s Conversational Engagement Based on Head Pose Information”, Proceedings of the 11th International Conference on Intelligent Virtual Agents (IVA), pp. 262–268, 2011.

35. Ryo Ishii, Shiro Ozawa, Harumi Kawamura, Akira Kojima, “MoPaCo: Hightelepresence video communication system using motion parallax with monocular camera”, Proceedings of IEEE International Workshop on Human-Computer Interaction: Real-time vision aspects of natural user interfaces, pp. 463–464, 2011.

36. Ryo Ishii, Shiro Ozawa, Takafumi Mukouchi, Norihiko Matsuura, “MoPaCo: Psudo 3D Video Communication System”, Proceedings of International Conference on Human-Computer Interaction (HCII), vol. 12, pp. 131–140, 2011.

37. Ryo Ishii, Yuta Shinohara, Yukiko, I. Nakano, Toyoaki Nishida, “Combining multiple types of eye-gaze information to predict user’s conversational engagement”, Proceedings of Workshop on Eye Gaze on Intelligent Human Machine Interaction (Gaze-in) , 2011.

38. Ryo Ishii, Hajime Noto, Hideaki Takada, Munekazu Date, Norihiko Matsuura, “Precise Distance Expression Using Motion Parallax in Video Communication”, Proceedings of International Conference on 3D Systems and Applications (3DSA), 2010.

39. Ryo Ishii, Yukiko, I. Nakano, “An Empirical Study of Eye-gaze Behaviors: Towards the Estimation of Conversational Engagement in Human-Agent Communication”, Proceedings of the workshop on Eye gaze in intelligent human machine interaction (Gaze-in), pp. 33–40, 2010.

40. Yukiko, I. Nakano, Ryo Ishii, “Estimating User's Engagement from Eye-gaze Behaviors in Human-Agent Conversations”, Proceedings of Intelligent User Interface (IUI), pp. 139–148, 2010.

41. Ryo Ishii, Yukiko, I. Nakano, “ Estimating User's Conversational Engagement Based on Gaze Behaviors”, Proceedings of International Workshop on Intelligent Virtual Agents (IVA), pp. 200–207, 2008.

42. Ryo Ishii, Toshimitsu Miyajima, Kinya Fujita, Yukiko I. Nakano, "Avatar’s gaze control to facilitate conversational turn-taking in virtual-space multi-user voice chat system", Proceedings of International Workshop on Intelligent Virtual Agents (IVA), pp. 458-458, 2006.

Book chapters

1. Ryo Ishii, Ryota Ooko, Yukiko, I. Nakano, Toyoaki Nishida, “Eye Gaze in Intelligent Human Computer Interaction”, Springer, pp. 85–110, 2013.

Technical article

1. Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Junji Yamato, "Prediction of "Who Will Be Next Speaker and When" in Multi-Party Meetings", NTT Technical Review, Vol. 13, No. 7, 2015.

Professional Activities

Committees

1. The ACM International Conference on Multimodal Interaction (ICMI). Program Committee Member, (2019, 2020)

2. The ACM International Conference on Multimodal Interaction (ICMI). Late-breaking Program Committee Member, (2020)

3. The ACM International Conference on Multimodal Interaction (ICMI). Demo Chair, (2016)

4. The ACM International Conference on Intelligent Virtual Agents (IVA), Program Committee Member, (2018)

5. The International Conference on Human-Computer Interaction (HCII), Session Organizer (Data driven social Machine Learning), (2019-2020)

6. ACHI Industry/Research Advisory Committee, (2017-present)

7. ACHI Industry/Research Technical Program Committee, (2016-present)

8. The IEICE, Society Journal Editorial Board Standing Review Committee, (2016–present)

9. The IEICE Transaction on Human Communication (HC) Special Issue, Editorial Committee, (2018–present)

10. The IEICE Human Communication Basic Research Society, Assistant secretary  (2016-2017)

11. The IEICE Human Communication Basic Research Society,  Expert advisor (2015, 2017-present)

12. The IEICE Verbal/Non-Verbal  (VNV) communication society, Steering Committee  (2014-present)

13. The Journal of Information Processing Society of Japan, Editorial board member, (2018–present)

14. Information Processing Society of Japan Symposium INTERACTION, Program Committee, (2017–present)

15. Multimedia, Distributed, Cooperative and Mobile (DICOMO) Symposium, Executive Committee, (2019)

16. The Japan Society for Artificial Intelligence (JSAI), “Social Signal Processing and AI”, Session Organizer (2017-2019)

17. The Journal of the Japanese Society for Artificial Intelligence (JSAI) (Special Issue on Dialogue System), Editor, (2018)

18. Transaction on Human Interface Society (HIS) (Special Issue on Human Collaborating), Deputy Editorial Chair, (2018)

19. Human Interface Symposium of Human Interface Society, Executive Committee, (2016) 

Reviewer

1. The Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing, (2019)

2. The ACM Conference on Human Factors in Computing Systems (CHI), (2017)

3. The ACM International Conference on Multimodal Interaction (ICMI), (2016, 2019, 2020)

4.  The International Conference on Advances in Computer-Human Interactions (ACHI), (2016--2020)

5. The IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), (2018)

6. Others, Journal and Conference Papers of The Institute of Electronics, Information and Communication Engineers, Information Processing Society of Japan, Artificial Intelligence Society, Human Interface Society, Journal of Electrical Society, Journal of Japan Society, etc.