End-to-end learning has shown promise in various tasks, such as computer vision and natural language processing. Recently, rapid progress has also been made in end-to-end robot learning for both high-level planning and low-level control. As a representative, the multimodal large model PaLM-E is able to decompose a long-horizon goal into many subskills, and the multi-task Robotics Transformer (RT-1) can directly output the low-level actions for each subskill, therefore forming a clear and concise pipeline and enabling a robot capable of performing a diverse set of tasks through only two end-to-end models.

Central to these astonishing achievements is the foundation of substantial training data in the field of robot learning. As part of the RT-X project at Google, we proudly unveil a comprehensive dataset, encompassing over 400 instances of robot demonstrations executed on intricate manipulation tasks. This resource represents a cornerstone for future strides in robot learning research, poised to catalyze breakthroughs and fuel innovation in the ever-evolving landscape of robotics.

This dataset can be used for vision-based robot imitation learning policies, fine-tuning visual representation models, training generative models that generate robot actions given task instructions, etc. It is collected with a FANUC Mate 200iD Robot, both RGB videos (third-person view and ego-centric view), robot trajectories (joint and action values), and language instructions are provided.R

Dataset format. The format is as follows.

• State 

• Main camera (fixed third-person view, 224 x 224 x 3)

• Wrist camera (ego-centric view, 224 x 224 x 3)

• Proprioceptive information (joint pose, gripper state, joint vel)

• Language instruction (for the task)

• Action

• Cartesian space actions

More detailed information about the data set format can be found  here. 

Download. Our dataset is included in Google's RT-X project and can be downloaded from here.

License. The dataset is released under the permissive MIT license, allowing free use, redistribution, and adaptation for non-commercial purposes.

BibTeX. 

@misc{zhu2023fanuc, 

  title={Fanuc Manipulation: A Dataset for Learning-based Manipulation with FANUC Mate 200iD Robot}, 

  author={Zhu, Xinghao and Tian, Ran and Xu, Chenfeng and Huo, Mingxiao and Zhan, Wei and Tomizuka, Masayoshi and Ding, Mingyu}, 

  howpublished={\url{https://sites.google.com/berkeley.edu/fanuc-manipulation}}, 

  year={2023} 

}