Demos


Unsupervised Feature Learning for 3D Scene Labeling

This video demonstrates combining HMP sliding window and HMP3D voxel features in an MRF framework for labeling objects in 3D scenes reconstructed from RGB-D (Kinect) videos. The top left shows the original RGB and depth video frames. The 3D scene labeling is shown as it is reconstructed from the Kinect video, with objects colored by category (sofa=maroon, coffee table=purple, bowl=red, cap=green, mug=yellow, soda=cyan).

Related Publications:

Unsupervised Feature Learning for 3D Scene Labeling 
Kevin Lai, Liefeng Bo, and Dieter Fox 
IEEE International Conference on Robotics and Automation (ICRA), May 2014. [PDF] [bibtex]
Finalist for Best Vision Paper Award. 


Object Labeling in 3D Scenes

In this video we demonstrate a view-based approach for labeling objects in 3D scenes reconstructed from RGB-D (Kinect) videos. The top row shows the original RGB and depth video frames, with high scoring bounding box object detections plotted on the RGB image. The 3D scene labeling is shown at the bottom, with objects color coded by category (bowl=red, cap=green, cereal=blue, mug=yellow, soda=cyan).


Related Publications:

Detection-based Object Labeling in 3D Scenes 
Kevin Lai, Liefeng Bo, Xiaofeng Ren, and Dieter Fox 
IEEE International Conference on Robotics and Automation (ICRA), May 2012. [PDF] [bibtex] [code]


OASIS (Object-Aware Situated Interactive System)

OASIS is a software architecture that enables the prototyping of applications that use RGB-D cameras and computer vision algorithms to recognize and track objects and gestures, combined with interactive projection. Object recognition is an important component of OASIS. The system recognizes objects that are placed within the interactive projection area so that the appropriate animations and augmented reality scenarios can be created. Our approach uses both depth and color information from the RGB-D camera to recognize different objects. The system can learn to recognize novel objects on the fly.

One example application of OASIS is the following interactive LEGO playing scenario that was shown at the Consumer Electronics Show (CES) 2011. In addition to object recognition, we also included pose estimation in this system. Taken together, this allows us to project the correct animations on the objects, including fire from the dragon's mouth, the road in front of the house, water from the fire truck, and the arrow for laying tracks in front of the train.


The video below demonstrates the object recognition system, including learning new objects on the fly:


Related Publications:

A Scalable Tree-based Approach for Joint Object and Pose Recognition 
Kevin Lai, Liefeng Bo, Xiaofeng Ren, and Dieter Fox 
Twenty-Fifth Conference on Artificial Intelligence (AAAI), August 2011. [PDF] [bibtex] [slides]