HMD Controller Dataset

News

Jul. 23rd, 2018: The training partition of the dataset has been uploaded and ready to use! It contains >500,000 image pairs for handheld object pose tracking. To get our dataset, please follow instructions in the Download section below.

Overview

We present a comprehensive image dataset of handheld controllers for AR and VR head-mounted display systems. Each image is annotated with the 6DOF pose of the controller in the camera space. Almost all images contain no visible markers for pose tracking.

Besides training dataset here, we have retained a testing dataset for an upcoming Kaggle competition. Subscribe to our maillist to receive updates on the competition!

Here are some sample groundtruth images from the dataset:

Dataset Details

The training partition of dataset contains >500,000 image pairs from 15 participants. Participants held a small controller in their hands during the data collection.

Each image is annotated with the 6DOF pose of the controller in the camera space. Annotations for images in the same folder are saved in labels.csv with the following format:

• left_image,right_image: filenames of the left and right images of the image pair.
• timestamp: Timestamp in milliseconds of the capture time. Timestamps are reset for each individual folder.
• gesture_label: Movement pattern for captures in the folder.
• width,height: Image width and height.
• x,y,z,qx,qy,qz,qw: 6DOF pose of the controller in left camera's space.
• left_intrinsics_fx,left_intrinsics_fy,left_intrinsics_cx,left_intrinsics_cy,left_intrinsics_w: left camera intrinsics.
• right_intrinsics_fx,right_intrinsics_fy,right_intrinsics_cx,right_intrinsics_cy,right_intrinsics_w: right camera intrinsics.
• extrinsics_x,extrinsics_y,extrinsics_z,extrinsics_qx,extrinsics_qy,extrinsics_qz,extrinsics_qw: extrinsics between the left and right camera.
• is_bad_frame: Flag on whether the frame may contain 6DOF tracking errors. Details of frame filtering can be found in our paper.

Images often contain cluttered background and very challenging lighting conditions.

Download

If you would like to download the dataset for research, please complete this short form to request access.

Citation

Egocentric 6-DoF Tracking of Small Handheld Objects

Rohit Pandey, Pavel Pidlypenskyi, Shuoran Yang, Christine Kaeser-Chen

Contact

For questions on the dataset, please contact us at hmd-controller-discuss@googlegroups.com. You may need to first join the group first through https://groups.google.com/d/forum/hmd-controller-discuss (click on "Apply for membership").