Detecting Object Affordances with Convolutional Neural Networks

Anh Nguyen, Dimitrios Kanoulas, Darwin G. Caldwell, and Nikos G. Tsagarakis

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016

Abstract:

We present a novel and real-time method to detect object affordances from RGB-D images. Our method trains a deep Convolutional Neural Network (CNN) to learn deep features from the input data in an end-to-end manner. The CNN has an encoder-decoder architecture in order to obtain smooth label predictions. The input data are represented as multiple modalities to let the network learn the features more effectively. Our method sets a new benchmark on detecting object affordances, improving the accuracy by 20% in comparison with the state-of-the-art methods that use hand-designed geometric features. Furthermore, we apply our detection method on a full-size humanoid robot (WALK-MAN) to demonstrate that the robot is able to perform grasps after efficiently detecting the object affordances.

Paper: PDF

Related Publications:

AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection

Thanh-Toan Do*, Anh Nguyen*, Ian Reid

ICRA 2018. (* equal contribution)

PDF Code

Object-Based Affordances Detection with Convolutional Neural Networks and Dense Conditional Random Fields

Anh Nguyen, Dimitrios Kanoulas, Darwin G. Caldwell, Nikos Tsagarakis

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017.

Video: