VRKitchen

An Interactive 3D Virtual Environment for Task-oriented Learning

About

Out goal is to enable better learning of autonomous agents for tasks with compositional goals and rich object state changes. To this end, we have designed VRKitchen, an interactive 3D virtual kitchen environment which provides a platform for training and evaluating various learning and planning algorithms in a variety of cooking tasks. With the help of virtual reality device, human users serve as teachers for the agents by providing demonstrations in the virtual environment.

Features

Physical Simulation

VR Interface

Python API

VR Chef Challenge

We propose the VR chef challenge consisting of two sets of cooking tasks:

Tool Use: requires an agent to continuously control its hands to make use of a tool.

Preparing dishes: agents must perform a series of atomic actions in the right order to achieve a compositional goal.

Environment Statistics

16 kitchens

4 characters

18 object state changes

5 dishes

Team

Xiaofeng Gao

Ran Gong

Tianmin Shu

Zhixiong Nan

Xu Xie

Shu Wang

Phipson Lee

Song-Chun Zhu

Download

Dataset

Code

Publications

VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning

Presents the VRKitchen platform and conducts experiments using RL algorithms on several virtual cooking tasks, for both high-level dish preparing and low-level tool use.

Gao, X., Gong, R., Shu, T., Xie, X., Wang, S., & Zhu, S. C. (2019).

arXiv:1903.05757.

Learning to infer human attention in daily activities

Presents a new video dataset collected on the VRKitchen platform, and proposes a deep neural network model that fuses both low-level human pose cue and high-level task encoding cue for inferring the task-driven inside-video human attention.

Nan, Z., Shu, T., Gong, R., Wang, S., Wei, P., Zhu, S. C., & Zheng, N. (2020).

Pattern Recognition, 107314.

Citation

@article{VRKitchen,
  author    = {Xiaofeng Gao and
               Ran Gong and
               Tianmin Shu and
               Xu Xie and
               Shu Wang and
               Song{-}Chun Zhu},
  title     = {VRKitchen: an Interactive 3D Virtual Environment for Task-oriented
               Learning},
  journal   = {arXiv},
  volume    = {abs/1903.05757},
  year      = {2019},
}

Press Coverage

Acknowledgements

The work reported herein is supported by DARPA XAI N66001-17-2-4029 and ONR MURI N00014-16-1-2007.