Embedded Files
About
About
Out goal is to enable better learning of autonomous agents for tasks with compositional goals and rich object state changes. To this end, we have designed VRKitchen, an interactive 3D virtual kitchen environment which provides a platform for training and evaluating various learning and planning algorithms in a variety of cooking tasks. With the help of virtual reality device, human users serve as teachers for the agents by providing demonstrations in the virtual environment.
Out goal is to enable better learning of autonomous agents for tasks with compositional goals and rich object state changes. To this end, we have designed VRKitchen, an interactive 3D virtual kitchen environment which provides a platform for training and evaluating various learning and planning algorithms in a variety of cooking tasks. With the help of virtual reality device, human users serve as teachers for the agents by providing demonstrations in the virtual environment.
Features
Features
Physical Simulation
Physical Simulation
VR Interface
VR Interface
Python API
Python API
VR Chef Challenge
VR Chef Challenge
We propose the VR chef challenge consisting of two sets of cooking tasks:
We propose the VR chef challenge consisting of two sets of cooking tasks:
Tool Use: requires an agent to continuously control its hands to make use of a tool.
Tool Use: requires an agent to continuously control its hands to make use of a tool.
Preparing dishes: agents must perform a series of atomic actions in the right order to achieve a compositional goal.
Preparing dishes: agents must perform a series of atomic actions in the right order to achieve a compositional goal.
Environment Statistics
Environment Statistics
16 kitchens
16 kitchens
4 characters
4 characters
18 object state changes
18 object state changes
5 dishes
5 dishes
Team
Team
Xiaofeng Gao
Ran Gong
Tianmin Shu
Zhixiong Nan
Xu Xie
Shu Wang
Phipson Lee
Song-Chun Zhu
Download
Download
Dataset
Dataset
Code
Code
Publications
Publications
VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning
VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning
Presents the VRKitchen platform and conducts experiments using RL algorithms on several virtual cooking tasks, for both high-level dish preparing and low-level tool use.
Gao, X., Gong, R., Shu, T., Xie, X., Wang, S., & Zhu, S. C. (2019).
Learning to infer human attention in daily activities
Learning to infer human attention in daily activities
Presents a new video dataset collected on the VRKitchen platform, and proposes a deep neural network model that fuses both low-level human pose cue and high-level task encoding cue for inferring the task-driven inside-video human attention.
Nan, Z., Shu, T., Gong, R., Wang, S., Wei, P., Zhu, S. C., & Zheng, N. (2020).
Citation
Citation
@article{VRKitchen,
author = {Xiaofeng Gao and
Ran Gong and
Tianmin Shu and
Xu Xie and
Shu Wang and
Song{-}Chun Zhu},
title = {VRKitchen: an Interactive 3D Virtual Environment for Task-oriented
Learning},
journal = {arXiv},
volume = {abs/1903.05757},
year = {2019},
}
Press Coverage
Press Coverage
Acknowledgements
Acknowledgements
The work reported herein is supported by DARPA XAI N66001-17-2-4029 and ONR MURI N00014-16-1-2007.
The work reported herein is supported by DARPA XAI N66001-17-2-4029 and ONR MURI N00014-16-1-2007.
Google Sites
Report abuse