Home

SMILEE: Symmetric Multi-modal Interactions with Language-gesture Enabled (AI) Embodiment

Sujeong Kim, David Salter, Luke DeLuccia, Kilho Son, Mohamed R. Amer and Amir Tamrakar

SRI International, Center for Vision Technologies, Princeton, New Jersey

We demonstrate an intelligent conversational agent system designed for advancing human-machine collaborative tasks. The agent is able to interpret a user's communicative intent from both their verbal utterances and non-verbal behaviors, such as gestures. The agent is also itself able to communicate both with natural language and gestures, through its embodiment as an avatar thus facilitating natural symmetric multi-modal interactions. We demonstrate two intelligent agents with specialized skills in the Blocks World as use-cases of our system.

We invite you to play with SMILEE!

Click HERE to Go to PlayWithSMILEE webpage.

Paper (pdf)

Supplementary Videos

<Main Video ( 2 minutes )>

Introduction to the BW apparatus and BW Conversational agent system
Symmetry Game Play in Virtual and Real world setup

<Uncompressed version of the main video>

<Blocks World Apparatus>

Block tracking
Initial version of Tower-building game in Blocks World

This work was funded by the Defense Advanced Research Projects Agency (DARPA) under agreement number W911NF-15-C-9244. The views, opinions, and/or conclusions contained in this paper are those of the author and should not be interpreted as representing the official views or policies, either expressed or implied of the DARPA or the DoD.

Google Sites

Report abuse