Design

Design Criteria

Play Any Notes: We should be able to input a list of musical notes in the range of one octave to the computer.
Compute Poses of Keys: The robot should accurately compute the physical coordinates of the spot to strike on each key, i.e. the key's center, regardless of where the piano is placed.
Good Sound: The robot should be able to strike a desired key from above, with enough force to produce an audible sound.
Play Continuously: The robot should be able to play multiple notes in succession, i.e. less than 10 seconds between each note, without looking at the scene again to re-evaluate.

Play Any Notes: Input notes as strings containing the note name and the octave number, e.g. "A1" or "C2#".
Compute Poses of Keys: Place two small AR tags at fixed positions on the keyboard, one near the bottom, one near the top. Use the fixed dimensions of the piano to compute the offset of each key to the closest AR tag. Convert the poses in the AR frame to the poses in the world frame.
Good Sound: Create a gripper extension that can hit the key without bending or sliding. Set waypoints to ensure the keys are struck directly from above.
Play Continuously: Compute all poses by calibrating with the AR tags. As long as the piano is not moved, these will remain constant throughout a series of notes.

Instrument Choice

Our initial plan was to make the robot play a xylophone, but we discovered two issues:
- The keys of the xylophone are too slippery. This low traction make the robot gripper slip, producing a bad sound.
- To produce a good sound on the xylophone requires a very quick up and down motion, which is impossible with the native Baxter gripper. We would have to build a custom spring-loaded mechanism.
These problems are even worse on a real piano keyboard, since keys are adjacent so slipping can result in playing the wrong note.
We switched to a roll-up piano with silicon rubber keys to solve these issues:
- The rubber offers the fingers more traction, preventing slipping.
- Slowly pressing down on a key produces a good sound, so a quick up and down motion is not required.

Method for Computing Poses of Keys

Computer vision strategy:
- Compute the pixel positions of each key using computer vision.
- Compute the homography matrix mapping pixel positions to physical poses.
- This strategy has the potential to need no AR tags, which would make it more robust; however, we tried to manually compute a 4 point homography by getting the pixel centers and physical poses of 4 AR tags.
- When implementing this strategy, we ran into complications, discussed here, so we decided to try the more viable AR tag strategy.
AR tag strategy:
- As described in the "Design We Chose" section, compute the fixed offsets from each key to an AR tag.
- At first, we used one large AR tag on the right side of the piano.
- For increased accuracy, we switched to two small AR tags. The pose of each key can be computed relative to the closest AR tag, so the average distance from AR tag to key is shorter. This means there is less room for measurement error to build up.
- This strategy was simple and sufficiently reliable, so we went with it in our final design.

Robot Choice

We had to decide between Sawyer and Baxter:
- Sawyer is more precise, with 0.1 mm task repeatibility compared to Baxter's 5 mm.
- Baxter has two arms, which would allows us to play a melody in one hands and chords in the other. This also makes detecting two small AR tags at different locations much easier, since we can use both arm cameras.
Since accuracy of the notes played was sufficient with Baxter, we went with Baxter.

Robustness: The requirement of playing one song continuously means our system is not robust to the piano moving during the song. In an industry setting, this problem could be addressed by online computer vision; for example, computing which key the gripper is above by detecting the keys as the gripper passes over them.
Durability: Using a custom gripper extension made the hand playing the melody melody very durable; it consistently hit the key and produced a good sound. The hand playing the chord used a default gripper, which sometimes hit the keys at the wrong hit and produce a bad sound.
Efficiency: For computing poses of keys, the computer vision strategy required using 2 AR tracking packages and 4 AR tags, making it very inefficient. The AR tag offset strategy was much more efficient because we only needed 1 package and 2 tags.

Report abuse