Generative Modeling meets HRI

In recent years, the rapid evolution of generative modeling, encompassing Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and Diffusion Models, among others, has reshaped the landscape of image synthesis, enabling the generation of highly realistic and diverse visual content. Inspired by this progress, our workshop proposal seeks to explore the potential of applying generative modeling techniques to enhance human-robot interaction (HRI). We aim to gather the communities of robot learning and Human-Robot Interaction to discuss cutting-edge generative model techniques, modeling of human behaviors and robot motions, and opportunities to use them to achieve more intuitive human-robot interactions for robot teaching, data collection, language-base interactions and collaborative execution. 

Why are generative models important for research in HRI? HRI will benefit greatly from powerful large models that bring open-world knowledge and generalization to the classic HRI interaction workflows. Just as ChatGPT has become of popular use for non-technical users, it’s only a matter of time before these types of large models with vision and language capabilities will play a key role in generating and mediating interaction between humans and robots in daily life settings (home robots learning your home tasks from examples) and industrial deployments (co-bots in manufacturing). Generative models are also key for the creation of simulation environments (3D assets, scenes, tasks, language commands, and language-based task generation), and simulation environments are useful for data collection of human demonstrations, data generation, and policy training. It’s important for HRI researchers to foster collaborations that investigate how multi-agent interactions and human-like behaviors will play a role in these systems, whether in simulation or real settings. 

Why is HRI important to research in generative models? Conversely, HRI is pivotal for advancing research in generative models. Human interaction and feedback are essential for producing high-quality data for learning and value-aligned training. For example, reinforcement learning from human feedback (RLHF) has demonstrated significant advancements in model performance, enabling ChatGPT’s performance to surpass models learned from static language datasets. Generative models applied to robotics are fundamentally tied to human interaction. In data collection pipelines, we need to provide users with tools, methods, and interfaces to provide and curate high-quality data that can be used by learning algorithms. For model improvements, we need human feedback in the loop with policy learning iterations of fine-tuning during deployment. These are all core interaction problems that are studied in HRI and are now prime to be used in the loop with generative AI in both training and inference, bringing the knowledge from interactions and human-centered modeling into robot learning. 

Motion and Behavior Modelling and Generation 

• Generative modeling of human-like behaviors in imitation learning

• Generative modeling of valid robot motion plans and TAMP

• Generative modeling of human-robot interactions

• Imitation learning and learning from demonstrations (motion and tasks) 

• Imitation of multi-agent collaborative tasks 

• Diffusion Models for motion and behavior generation

• Generation of scenes, tasks, and interactive behaviors in simulation

Human Interaction for Goal Specification

• Interfaces for robot teaching

• Teleoperation and shared autonomy 

• User goal specification for interactively commanding a robot

Large Language Models (LLMs) and Vision Language Models (VLMs) in HRI

• LLMs and VLMs for embodied AI

• Generative models (LLMs/VLMs) for offline evaluation

• Generative models of speech for HRI (dialogue, empathy, engagement)

• LLMs as planners for behavior generation.

AI-HRI Safety and Alignment 

• Risks and biases of using generative models for data generation, interaction

• Safely deploying generative models for HRI

• Out-of-distribution settings in HRI