ICub Tries to Play an Electronic Keyboard

One of the most demanding cognitive challenges for a human mind is music performing. This project provides the iCub with a unique opportunity to interact with the environment. The robot is able to compute the 3D location of each electronic keyboard key it can see, listen to a sequence of musical notes a human played on the keyboard, and press the same musical notes in the same order on the keyboard. The design consists of three modules: a vision module, a pitch detection module, and a motor control module.

Vision Module

The vision module processes the visual input received from the iCub camera and obtain the 3D location of each keyboard key in the robot's sight. We do not use a set of predefined key locations. No markers are attached to the keys to giving any visual clues. The 3D location for each key can be obtained using the geometric relationship.

Pitch Detection Module

The pitch detection module detects a musical event and tells which musical notes were played. Given a wave file, an onset detection is first performed, so that the program knows when a new musical note is played. Then for each musical note, the YIN pitch detection algorithm is used for detecting the frequency of the note.

Motor Control Module

The vision module computes the 3D location for each key in the robot's view. The pitch detection module finds which keys are pressed by a human. The motor control module sends commands to arm joints such that the iCub can move his arms and hands to the desired places using inverse kinematics.

Demo

The video below shows that the robot presses C₅ after C₅ is heard by the robot: