> Design Projects > HOLO-PIANO
OVERVIEW
An AR piano remote coaching and training platform
Learning piano as an amateur could be effort taking considering the commuting back and forth from the coach’s place, which may take a huge part of one day's leisure time. Yet a remote session in the form of video call will lose a lot of essential information compared with side by side tutoring.
We explored the features of Hololens and MIDI, under the notion of Augmented Reality and Groupware to build a remote teaching and training platform, which connects both piano coach and learner. On the platform, the student learns to play piano remotely with the sense of coach's presence and practice by him/herself with the help of AR instructions.
KEYWORDS: AR, HoloLens, remote collaborative training
COURSE: MR&TUI, Groupware and Collaborative work
DURATION: Nov.2019 - Feb.2020
MY ROLE: User research, UX/UI design
RESEARCH
We carried out our reseach regarding three aspects, the know-how of current remote piano classes, the existing AR piano coaching software and the piano learners.
Skype piano class observation
Screenshot from Youtube
There are already many piano coaches carrying out classes online. By learning about the common practice of current remote teaching scenario, we get basic knowledge of what the teachers and students are usually equipped with, and how the classes are carried out. And by obsearving the recorded classes, we can also discover the focus of the coach during the sessions, and also break points the coach met. These could be served as our design guides and inspiration.
One of the most commenly used platform is Skype. During a typical Skype piano session, the coach will use two cameras at least, one installed above the keyboard and looking directly down onto it, and the other one on top of the computer screen facing the coach. The student uses only one camera and usually placed from aside so that the coach can see both student's upper body and the keyboard.
As Skype does not support duo camera under one account(and so do most of the video call softwares), the coach has to switch between views repetitively while teaching so that the student can get an overall view of the demo. Meanwhile, various mounting positions of the students' cameras also have impacts on the coach's instruction. The coach can only correct the students' hand gesture with verbal instruction or overhead-view instruction and this would often lead to waste of time.
Music Everywhere(Das, S., Glickman, S., Hsiao, F.Y. and Lee, B., 2017) is a HoloLens software that facilitates keboard self-teaching and improvising by placing virtual bars stemming from the roots of the keys. In order to confront with the issue of occlusion, since the image "projected" onto the keys are actually display in HoloLens, the team introduced Leap Motion, attached on HoloLens, to capture user's hand and mask out the image correspondingly. This inspired us that a Leap Motion might be used to transfer a teacher's real-time demo onto the user's keyboard.
Product Research
Music Everywhere
User Research
interview with piano player in Massy, France
We carried out the user research by interviewing and dispatching questionnaires to some of our potential users who are learning piano, and other proxies such as skilled players who have learned piano, and their parents who had witnessed them learning the instrument.
Novice users need more time, attention and practice to properly play the piano. A few users mentioned that they initially struggled to play the piano with both hands and have proper coordination. Many users had classes once a week and didn’t feel like they were learning much in that 2 hr session.
While learning, the most important thing reported by several piano learners is the hand position and the elbow movement when changing keys, which users usually couldn’t figure out by themselves when they followed videos of instructors. Remote learners, especially those who practice and learn from pre-recorded videos and tutorials lacked a music learning environment to immerse themselves fully into music and didn’t get any feedback. Most of the piano learners would like to improvise songs and want to learn the technique of improvising.
CRUCIAL USER FINDINGS
1. Fingering is important, both the orders of fingers and the position when crossing fingers. Beginners often get lost in fingering and make mistakes in initial stages. A close-up view of coach’s hands is crucial for learners, either being remote or on the spot.
2. For learners, a piece of continuously played melody is rewarding yet requires a lot of practice. Coordination of two hands is hard for beginners and often results in discouragement. The technique of accompaniment play is often adopted by the coachs in order to help students catch the tempo.
SYSTEM DIAGRAM
According to the user findings, we designed a system using HoloLens to project coach’s hands and key pressed onto the learner’s keyboard. The virtual hands are captured by Leap Motion. Meanwhile, both can also see each others’ upper body.
USER SCENARIOS
1 Setup
Using Vuforia SDK, we assigned a little standing pad printed with HOLO-PIANO logo as marker so that Hololens can map the corresponding UI onto the musical note, MIDI and certain position in space. Idealy the marker is fixed on Piano. Since we don’t have an 88-key MIDI, we use the 25-key instead and in unity, the relative positions have already been claimed.
A dashed square in HoloLens will guide user to aim at the marker. Once the marker is recognised, the app will launch itself.
2 Remote teaching
Coaching contains two parts:
Demonstration: the coach plays and the virtual hands will float over learner’s keyboard hence the fingering can be observed by the learner in a close-up view.
Accompanying: the coach plays along with the learner so that the coach can give a tempo and help the learner establish an initial sense of the tune.
3 Self-practicing
The note is encoded in XML and user can practice with the instruction of corresponding dropping bars over the keys. In addition, certain sections can be selected and practiced repeatedly.
PROTOTYPES
Download the Principle demo on your mac
Fast paper prototype
EVALUATION AND CONCLUSION
We evaluated our prototype on semester project exhibition. The idea was generally praised as appealing and unique. Yet due to the complexity of communication network, the experience was influenced by the accumulated delay in signal transmission.
As a tech-oriented design project, the business feasibility of HoloLens deployment at users' was somehow not thoroughly discussed. It is obvious that it's not the best way to let the piano learners purchace such device while subscribing to a piano class. Possible models could be that, 1) individual coach subscribes the device from the platform and add the cost into coaching tuition as added value of their innovative classes, 2) the piano teaching institute purchase or licence the devices and dispatch to students.
By the end of the project, the global pandemic burst out and profoundly changed our life. More events are hosted online and remote education and tutoring has seen the biggest surge throughout the years. Given the condition granted, we are looking forward to our design’s broader impact. In one way we can replicate and even enhance the experience of conventional piano coaching, and in another way we would happily see if more coaches’ business influenced by the social distancing will get back on track with a new modality norm.
