Jam On! - Music for the Visually Impaired

Zach Baker | Michael Belch | Isabella Hsia | Kiana Kazemi

Anurag Kulkarni | Fredrick Ladauti | Antoinette Walter | Maximilian Warias

This project was developed as part of ENGR 017 The Art of Making: An Introduction to Hands-On System Design and Engineering, a course taught at the University of Pittsburgh's Swanson School of Engineering.

First, we'd like you to imagine.

Have you ever played a musical instrument?

Visualize it - (and if you're not musically inclined, pretend you play the harmonica)

Picture your hands on the keys. Your fingers on the strings.

Now imagine playing your favorite instrument … but you can't see.

This is a reality for the vast majority of visually impaired musicians.

Why is this important?

According a 2009 study conducted by researchers at the Institute of Education in London, 90% of blind children showed interest in learning music, as opposed to 67% of children with low sight, and 38% of sighted children. That's 65 million visually-impaired children worldwide—so why do we rarely see blind musicians in practice?

The system isn't catered to them!

Our Problem Statement

━━━ ♬ ━━━

Young students who are visually-impaired struggle to gain a strong foundation in music because the current music education system caters too much to the able and lacks individualization.

Who are we designing for?

We are focused on helping young children (ages 6-10 years old) who are visually impaired.

Specifically, we are helping those who are interested in learning about or playing music.

User Outreach + Main Takeaways

We reached out to a total of 71 experts, potential users and stakeholders, conducting interviews both in-person and remotely.

Here were some of the things we learned.

Just like visually impaired individuals rely on braille to read text, visually impaired musicians rely on braille music code (BMC) to read music. BMC is not the same as braille, and the learning curve is steep!

Methods of teaching music that involve a kinesthetic element—having to do with movement or sensation—like the Suzuki method, or education via eurhythmics, are especially important for young children. Music doesn't have to be just an auditory experience!

Individuals with visual impairments excel at learning music because they have a phenomenal inner ear. After doing some research, we found out that blind children are four thousand times more likely to have perfect pitch than their fully-sighted peers.

Drumroll, please...

After months of ideating, building, and testing, these are our solutions.

JamBox

Users are able to click on each button which will play a specific musical note. There is braille music code on each tile to help the user associate the specific tone with the note name. The shapes, colors, lights, and reflective tape one each button help the user better distinguish between buttons.

mimD (Music in Motion Detector)

mimD uses skeleton tracking to help users make music. Users can move their hands and body up and down. As they move, musical notes will be played based on their body's positioning. As the user moves their body up, higher notes are played, and as they move down, lower notes are played.

Past Iterations

Our first mimD pretotype*, with a camera made out of foam core and a speaker made out of foam block.

The second iteration of mimD with the simulated camera and speaker combined. A removable LED strip is attached to the box via velcro.

Our first JamBox pretotype, fashioned out of a cardboard box and strips of velcro for modularity. The "keys", made out of foam core, were labelled with BMC.

The second iteration of JamBox made out of foam core. We implemented a locking mechanism to keep the keys in place, as well as diversity in key shapes and colors. The texture board on the left was comprised of 9 different textures, which we brought to testing.

*Pretotype was a word first coined by former Google executive Alberto Savoia in his book, "Pretotype It". Pretotyping refers to a methodology of testing an idea quickly and inexpensively by creating extremely simplified, mocked, or virtual versions of that product to help validate the premise that "If we build it, they will use it." As you can see, our initial pretotypes were extremely rudimentary and came together in less than 30 minutes. We were able to simulate functionality using synthpads and our computers, thereby allowing us to test our idea first before testing our product.

Discontinued Designs

One of the first pretotypes we took to testing was a BMC tuner known as the Bratrix. A vast majority of visually impaired children have phenomenal pitch perception, and the Bratrix was intended to help them tune their pitch using "pop up" braille. After testing with it, we realized it was a little too complicated for our target age range, and with heavy hearts, we discontinued it.

Testing

━━━ ♬ ━━━

To better understand our users and their current music learning environment, we sat in on classes at the Western Pennsylvania School for Blind Children (WPSBC). From there, we designed and tested our prototypes with a total of 19 classes (13 hours) across 65 students. We went through 3 total iterations of designs for the JamBox and mimD between testing sessions, which led us to our current designs.

JamBox Feedback

Key JamBox takeaways from testing were:

To implement variability in key shapes, colors, and textures
To use bright lights for low-vision students, as a vast majority of visually impaired students aren't completely blind and benefit from the visual stimulation
To make the box light and durable, so it would be both comfortable for and safe from the children

━━━ ♬ ━━━

mimD Feedback

Key mimD takeaways from testing were:

To allow for variability in movement, as most children who visually-impaired also have motor deficits that make it difficult to move
To implement an LED strip that would function similarly to the speaker, so low-vision students could better correlate their movement with the music
To bias towards softer sounds that stayed within a comfortable range of pitches

System Diagrams: JamBox

Top view of one of the JamBox buttons.

Bottom view of one of the JamBox buttons.

System diagram for JamBox.

System Diagrams: mimD

System diagram (code flow) for mimD.

Design Goals

━━━ ♬ ━━━

When thinking about our ultimate goals for this project, we wanted to:

Help kids build connections between their actions and movements and the musical feedback from our systems.
Make sure our systems are enjoyable and accessible to our test users.
Ensure that the kids are able to engage with our designs.

Results

JamBox

Connection

40% of kids tested were able to effectively connect their actions to the musical feedback, falling short of the 70% goal we set. We hope to continue working to meet this goal in the future.

Enjoyability

52% of kids tested enjoyed using JamBox and found it engaging, again falling short of the 70% goal we set. We will also work to meet this goal in the future.

Accessibility

89% of kids tested freely interacted with the JamBox, meeting our threshold of 70% for kids tested.

━━━ ♬ ━━━

mimD

Connection

40% of kids tested were able to effectively connect their movements to the musical feedback, falling short of the 70% goal we set. We hope to continue working and meet this goal in the future.

Enjoyability

75% of kids tested enjoyed using mimD and found it engaging, meeting the 70% goal we set.

Accessibility

95% of kids tested freely interacted with the mimD, meeting our threshold of 70% for kids tested.

Future Plans

JamBox

For the JamBox, as we continue to work, we hope to

Add a variety of instrument sounds.
Add a game feature that helps kids learn notes by pitch.
Increase the durability and ergonomics of the system.

mimD

In the future, we hope to:

Add a variety of instrument sounds.
Increase the sensitivity to movements.
Detect a wider variety of movements, such as a head tilt.

Closing Remarks

━━━ ♬ ━━━

While we are proud of the work we have done thus far, we recognize that there is still much to be done. We hope to continue working to make our goals a reality and create a future in which music is more accessible for everyone.

Acknowledgements

━━━ ♬ ━━━

Our sincerest gratitude to Amber Haer and the kids at the WPSBC.

Thank you to our PFAs: Oday Abushaban, Sophia Buda, Samsher Sidhu, as well as our lovely professor, Dr. Samosky.

We owe our success to you most of all.

Page updated

Google Sites

Report abuse