Zach Baker | Michael Belch | Isabella Hsia | Kiana Kazemi
Anurag Kulkarni | Fredrick Ladauti | Antoinette Walter | Maximilian Warias
This project was developed as part of ENGR 017 The Art of Making: An Introduction to Hands-On System Design and Engineering, a course taught at the University of Pittsburgh's Swanson School of Engineering.
Have you ever played a musical instrument?
Visualize it - (and if you're not musically inclined, pretend you play the harmonica)
Picture your hands on the keys. Your fingers on the strings.
Now imagine playing your favorite instrument … but you can't see.
This is a reality for the vast majority of visually impaired musicians.
According a 2009 study conducted by researchers at the Institute of Education in London, 90% of blind children showed interest in learning music, as opposed to 67% of children with low sight, and 38% of sighted children. That's 65 million visually-impaired children worldwide—so why do we rarely see blind musicians in practice?
The system isn't catered to them!
━━━ ♬ ━━━
Young students who are visually-impaired struggle to gain a strong foundation in music because the current music education system caters too much to the able and lacks individualization.
We are focused on helping young children (ages 6-10 years old) who are visually impaired.
Specifically, we are helping those who are interested in learning about or playing music.
We reached out to a total of 71 experts, potential users and stakeholders, conducting interviews both in-person and remotely.
Here were some of the things we learned.
Just like visually impaired individuals rely on braille to read text, visually impaired musicians rely on braille music code (BMC) to read music. BMC is not the same as braille, and the learning curve is steep!
Methods of teaching music that involve a kinesthetic element—having to do with movement or sensation—like the Suzuki method, or education via eurhythmics, are especially important for young children. Music doesn't have to be just an auditory experience!
Individuals with visual impairments excel at learning music because they have a phenomenal inner ear. After doing some research, we found out that blind children are four thousand times more likely to have perfect pitch than their fully-sighted peers.
After months of ideating, building, and testing, these are our solutions.
Users are able to click on each button which will play a specific musical note. There is braille music code on each tile to help the user associate the specific tone with the note name. The shapes, colors, lights, and reflective tape one each button help the user better distinguish between buttons.
mimD uses skeleton tracking to help users make music. Users can move their hands and body up and down. As they move, musical notes will be played based on their body's positioning. As the user moves their body up, higher notes are played, and as they move down, lower notes are played.
Our first mimD pretotype*, with a camera made out of foam core and a speaker made out of foam block.
The second iteration of mimD with the simulated camera and speaker combined. A removable LED strip is attached to the box via velcro.
Our first JamBox pretotype, fashioned out of a cardboard box and strips of velcro for modularity. The "keys", made out of foam core, were labelled with BMC.
The second iteration of JamBox made out of foam core. We implemented a locking mechanism to keep the keys in place, as well as diversity in key shapes and colors. The texture board on the left was comprised of 9 different textures, which we brought to testing.
*Pretotype was a word first coined by former Google executive Alberto Savoia in his book, "Pretotype It". Pretotyping refers to a methodology of testing an idea quickly and inexpensively by creating extremely simplified, mocked, or virtual versions of that product to help validate the premise that "If we build it, they will use it." As you can see, our initial pretotypes were extremely rudimentary and came together in less than 30 minutes. We were able to simulate functionality using synthpads and our computers, thereby allowing us to test our idea first before testing our product.
One of the first pretotypes we took to testing was a BMC tuner known as the Bratrix. A vast majority of visually impaired children have phenomenal pitch perception, and the Bratrix was intended to help them tune their pitch using "pop up" braille. After testing with it, we realized it was a little too complicated for our target age range, and with heavy hearts, we discontinued it.
━━━ ♬ ━━━
To better understand our users and their current music learning environment, we sat in on classes at the Western Pennsylvania School for Blind Children (WPSBC). From there, we designed and tested our prototypes with a total of 19 classes (13 hours) across 65 students. We went through 3 total iterations of designs for the JamBox and mimD between testing sessions, which led us to our current designs.
Key JamBox takeaways from testing were:
To implement variability in key shapes, colors, and textures
To use bright lights for low-vision students, as a vast majority of visually impaired students aren't completely blind and benefit from the visual stimulation
To make the box light and durable, so it would be both comfortable for and safe from the children
━━━ ♬ ━━━
Key mimD takeaways from testing were:
To allow for variability in movement, as most children who visually-impaired also have motor deficits that make it difficult to move
To implement an LED strip that would function similarly to the speaker, so low-vision students could better correlate their movement with the music
To bias towards softer sounds that stayed within a comfortable range of pitches
Top view of one of the JamBox buttons.
Bottom view of one of the JamBox buttons.
System diagram for JamBox.
System diagram (code flow) for mimD.
━━━ ♬ ━━━
When thinking about our ultimate goals for this project, we wanted to:
Help kids build connections between their actions and movements and the musical feedback from our systems.
Make sure our systems are enjoyable and accessible to our test users.
Ensure that the kids are able to engage with our designs.
40% of kids tested were able to effectively connect their actions to the musical feedback, falling short of the 70% goal we set. We hope to continue working to meet this goal in the future.
52% of kids tested enjoyed using JamBox and found it engaging, again falling short of the 70% goal we set. We will also work to meet this goal in the future.
89% of kids tested freely interacted with the JamBox, meeting our threshold of 70% for kids tested.
━━━ ♬ ━━━
40% of kids tested were able to effectively connect their movements to the musical feedback, falling short of the 70% goal we set. We hope to continue working and meet this goal in the future.
75% of kids tested enjoyed using mimD and found it engaging, meeting the 70% goal we set.
95% of kids tested freely interacted with the mimD, meeting our threshold of 70% for kids tested.
For the JamBox, as we continue to work, we hope to
Add a variety of instrument sounds.
Add a game feature that helps kids learn notes by pitch.
Increase the durability and ergonomics of the system.
In the future, we hope to:
Add a variety of instrument sounds.
Increase the sensitivity to movements.
Detect a wider variety of movements, such as a head tilt.
━━━ ♬ ━━━
While we are proud of the work we have done thus far, we recognize that there is still much to be done. We hope to continue working to make our goals a reality and create a future in which music is more accessible for everyone.
━━━ ♬ ━━━
Our sincerest gratitude to Amber Haer and the kids at the WPSBC.
Thank you to our PFAs: Oday Abushaban, Sophia Buda, Samsher Sidhu, as well as our lovely professor, Dr. Samosky.
We owe our success to you most of all.