This assistive game device was made for our client Allie, a young girl with disabilities, as a part of CMU's Advanced Physical Computing course. The overall goal is to better our client's quality of life in some form, and through interactions with Allie, this device was made to give her a more independent yet collaborative playing experience. This device allows Allie to play a music game independently without outside assistance, and also to play collaboratively with others. The ideation and creation of this device depends on being tailored to Allie, which means spending a lot of time interviewing Allie and her surrounding people during project ideation and prototyping. 

The Assistive Piano Game Device we made is a game/toy for Allie that incorporates her love for Taylor Swift, interacting with others, and the fun of playing a game independently. The device looks similar to a double-sided piano. One side has 3 large keys for Allie, while the other side has 7 smaller keys for another player. The rules of the game are as follows: press the key that lights up within the designated time for music to start or continue playing. If the players do not press the lit up key in time or press the wrong key, the music stops and resets. There are also three setting adjustments on the side of the device: a mode switch, a volume knob, and a song button. The mode switch allows the device to change between solo player and multiplayer (in solo player, the keys light up only on Allie's side, and in multiplayer, the keys light up on both Allie and the other player's side). Then, the volume knob changes the volume of the music played. Finally, the song button changes the song that plays from the available songs in the device. On the bottom of the device, there is a power switch that powers on the device, and is powered by 4 replaceable AA batteries.

This process was made possible through many iterations of prototyping and meeting with Allie, which has been described in the "First Interview Documentation", "First-Round Prototype Documentation", and "Second-Round Prototype Documentation". After showing Allie our second-round prototype, we had quite a few modifications to make to the device to make it more finalized and seamless to use. This includes adding more features to the device to give more complexity to the device as well as ease of use for Allie. 

This sketch shows how the physical design moved forward based on what we learned from earlier prototypes. The overall curved shape was designed to reduce sharp edges and make the device safer to touch. It also creates a slight slope so Allie can better see the LED lights on the top. The keys were refined to move both upwards and downwards, making them easier to press with less effort. More rounded edges and chamfers were added to protect Allie’s hands during use. The black button placed between the keys follows a piano-like layout, but it also works as a physical divider to reduce accidental presses between neighboring keys. The sketch separates Allie’s larger keys from the regular keys and introduces two modes: a solo mode for independent play and a co-play mode for playing together with others.

The main changes made in the circuit design is the following:

• The microcontroller is switched from Raspberry Pi Pico to Teensy 4.1 (includes Teensy Audio Shield and SD card)

• Power source comes from batteries

• Additional other player keys, and LEDs to LED strips (7 extra keys)

• Add mode switch, change song button, volume control, and small updates in game logic

To add more features to the device, a mode switch (pink toggle switch) was added to give the device two options when playing: a solo player mode in which the keys would only light up on Allie's side, and a multiplayer mode in which the keys would light up on both Allie's side and the other player's side. Additionally, a song button was added to allow the user to shift between songs on the device (red push button). Additionally, to allow for a less confusing gaming experience, the next key picked to light up will not repeat to be the same one, to avoid confusion with whether the key has been pressed or not. Finally, we allow the user to control the volume by changing the potentiometer to be a bigger one and bringing it to the outside of the device to give the user access to the volume knob. An issue we ran into was that we did not have a 200kΩ potentiometer, which was what was recommended by the part. By using some testing with a 100 kΩ potentiometer with a 100kΩ resistor in series and a 250  kΩ potentiometer, the 250 kΩ potentiometer gave better results, which was what was used in the final device. Overall, the additional adjustments made the device more usable and customizable for the user, as well as a more finalized product.

According to the Gantt chart, we deviated quite a bit from the original schedule. We were a little behind from the planned schedule; however, the original plan was made to be early to account for the chance of falling behind. However, the code and electronics were able to be finished before the final critique on 12/4, but there were problems with the physical design being finished by then due to some issues with the 3D printers. However, we were able to get the project well finalized in advance before giving it to Allie. The main reason for diverging from our planned schedule was that we didn't know how long certain things took. For example, running into bugs with faulty hardware components and 3D printers. This made the timeline unpredictable. This experience helps us learn to front load our work even more to stay on schedule. 

During the final critique, participants gave enriching feedback and comments on our project. For example, one of the people invited to the discussion complimented that we "took into consideration many interests of the client, including specific song choices." This piece of feedback emphasized the importance of having a variety of song selections for Allie, and encouraged more care into the song choices put into the device. Since I had already emailed Allie's mother for Allie's favorite songs, confirmed which songs Allie likes in person during in-person interviews, and asked Allie's mother for her opinion on new songs for Allie, I made sure to add as many songs that I could for Allie, putting as many songs I knew that she liked. Additionally, another person said that they think "it is a great product for her to enjoy interacting with others." Since this device's multiplayer mode was made because of how much Allie likes spending time and interacting with her loved ones, this comment demonstrates that this goal of the device allowing for Allie to play for herself with her friends, caregivers, and family was met. 

One major takeaway from working on this project is the importance of tailoring the device to Allie. While this was something we actively tried to consider while making prototypes, there were aspects of the device we did not know to consider until we gave the device to Allie to test during prototype iterations. For example, the keys’ light being visible enough and there being guards in between the keys to give Allie's hand support. Another major takeaway was the importance of having Allie test out the device. Continuing with the previous takeaway, there was a lot to learn that could only really be learned with Allie testing the device out herself. Thus, with devices meant to tailor to a specific person, I think the best way to do it is to have the user the device is tailored for to test it themselves, and to have as many iterations as necessary. Finally, one last major takeaway is that making a device tailored to a person also requires understanding them beyond what physically works. For example, I think that understanding their daily life, preferences, the people around them, etc. are necessary to make a tailored device. For example, understanding the context of Allie at home, at after school, and what things are lacking in her life in which the device can help fill the gap, which, in this case, is a device that allows her to independently play, and also with others. 

Ultimately, I am very proud of this project, and that at the end of the day Allie has a device in which she can use to play with, both individually and with others. There is always room for improvement, however. If given more time, I would incorporate some great ideas people had during the final critique. For example, someone said to add “SD card access to add more songs” and to make “the length of time before [...] song stoppage” changeable “if [Allie] wants it to be harder or easier on a given day based on how she is feeling." I think both are great ideas, but have some technical challenges and require more time than the time we have to revise the device to give to Allie. However, at the end, there are still a lot of adjustable parts of the device that hit the main points we wanted the device to have. 

-- Sylvia Ker

This project felt very different from a typical studio project because we were designing for a real person instead of an imagined user. Working directly with Allie made it clear very quickly that many ideas that sound reasonable in theory don’t always work in practice. Things like button size, how much force was needed to press a key, and how the keys were spaced ended up being much more important than we initially expected. Testing with Allie helped us see where our assumptions were wrong and pushed us to constantly adjust the design based on her actual interactions rather than our own expectations.

One piece of feedback from the final crit described the project as a “great idea and finely tuned to Allie,” which felt meaningful to us because a lot of our design decisions came directly from observing her preferences and physical abilities. Another comment highlighted that the “two-player mode is brilliant,” which reinforced our belief that designing for shared interaction was just as important as supporting solo play. Seeing this feedback helped confirm that the co-play mode was not just an added feature, but a core part of the project’s value.

We also received feedback that the “lighting design is superb,” which aligned with our intention to use light as a clear and engaging cue rather than just decoration. Lighting became a key way to guide interaction and signal timing, especially for maintaining attention during play. In addition, comments noting the effort to solder components “for better durability” reminded us that robustness matters just as much as interaction design when building for real-world use. These responses made us more aware that accessibility is not only about interaction logic, but also about how well a device holds up physically over time.

On a more personal level, what surprised me most was how much small physical changes could improve the overall experience. Simple adjustments like rounding sharp edges, increasing key size, or spacing keys more clearly made the instrument feel noticeably safer and easier to use. Through repeated testing, I learned that accessibility is rarely about one big solution. Instead, it comes from carefully refining many small details through iteration and feedback.

Looking back, I wish we had more time to keep refining the design. While the final prototype feels like a solid step, there are still many aspects of the mechanical structure, lighting behavior, and overall interaction that could be improved with more testing. With additional time, I think the project could become more robust and polished, especially in terms of long-term durability and flexibility for different play conditions. Overall, this project taught me the importance of designing with, rather than for, a specific person, and how valuable real feedback is in shaping meaningful and thoughtful physical interactions.

-- Zimeng Ma

/*

Code for Assistive Piano Game Device

...............................................................................

Sylvia Ker

Last Updated: 12/09/2025

...............................................................................

Code description:

Final Code for Assistive Piano Game Device. Code controls circuit of buttons,

LEDs, and audio. The logic of the game is as follows: the key that needs to be

pressed will light up with the according LEDs, if the user presses the key

within the designated time frame, the music will start/continue playing. If

not, the music will stop playing and restart. There is also a mode switch that

has two modes: solo and multiplayer, in which the keys that will light up vary

from only one side to both sides of the Assistive Piano Game Device. The is

also a song button that changes the song that plays.

...............................................................................

Pin mapping:

Teensy 4.1 Pin(s) | I/O    | purpose

-----------------------------------------

0-23              | output | Audio Shield

24                | output | audioLedPin: LED (on when Audio on)

25                | output | KEYLED_PIN: Key LED Strip

27                | output | TOPLED_PIN: Top LED strip

28                | input  | songPin: Change song button

29                | input  | modePin: Mode switch

30                | input  | buttonPins[0]: Allie key 1

31                | input  | buttonPins[1]: Allie key 2

32                | input  | buttonPins[2]: Allie key 3

35                | input  | buttonPins[9]: Player key 7

36                | input  | buttonPins[8]: Player key 6

37                | input  | buttonPins[7]: Player key 5

38                | input  | buttonPins[6]: Player key 4

39                | input  | buttonPins[5]: Player key 3

40                | input  | buttonPins[4]: Player key 2

41                | input  | buttonPins[3]: Player key 1

...............................................................................

Credit:

This code was written with the help of:

ChatGPT, Audio > WavFilePlayer.ino from Built-In Examples,

and Blink.ino from FastLED Custom Library.

*/

#include <Arduino.h>

#include <Wire.h>

#include <SPI.h>

#include <SD.h>

#include <Bounce.h>

#include <Audio.h>

#include <SerialFlash.h>

#include <FastLED.h>

//For users to change:

//.............................................................................

float playerTime = 1.0; //change difficulty (time for player to press button)

float allieTime = 10.0; //change difficulty (time for allie to press button)

//

//File name needs to be in SD card

//file name must be 44.1kHz sample rate and 16 bits per sample

//file name must be 8 characters max and all captialized

const char* songList[] = {

 "CRUELSUM.WAV", //Cruel Summer - Taylor Swift

 "SHAKEITO.WAV", //Shake it Off - Taylor Swift

 "LOVESTOR.WAV", //Love Story - Taylor Swift

 "BLANKSPA.WAV", //Blank Space - Taylor Swift

 "ANTIHERO.WAV", //Anti Hero - Taylor Swift

 "PARTYINT.WAV", //Party in The U.S.A. - Miley Cyrus

 "THECLIMB.WAV", //The Climb - Miley Cyrus

 "WRECKING.WAV", //Wrecking Ball - Miley Cyrus

 "ISEETHEL.WAV", //I See the Light - Tangled

 "AMILLION.WAV", //A Million Dreams - The Greatest Showman

 "PARTOFYO.WAV", //Part of Your World - Little Mermaid

 "HOWFARIL.WAV"  //How Far I’ll Go - Moana

};

const int numberofsongs = 12; //change the number of songs

int songIndex = 0; //the Nth song that will play by default

//.............................................................................

// Audio

AudioPlaySdWav playWav1;

AudioOutputI2S audioOutput;

AudioConnection patchCord1(playWav1, 0, audioOutput, 0);

AudioConnection patchCord2(playWav1, 1, audioOutput, 1);

AudioControlSGTL5000 sgtl5000_1;

// SD Card

#define SDCARD_CS_PIN BUILTIN_SDCARD

// Key LED strip

#define KEYLED_NUM 28

#define KEYLED_PIN 25

CRGB leds[KEYLED_NUM];

//LED ranges for light up

const int allie1Range[] = {0, 2};

const int allie2Range[] = {6, 7};

const int allie3Range[] = {11, 13};

const int pl1Range[]    = {14, 15};

const int pl2Range[]    = {16, 17};

const int pl3Range[]    = {18, 19};

const int pl4Range[]    = {20, 21};

const int pl5Range[]    = {22, 23};

const int pl6Range[]    = {24, 25};

const int pl7Range[]    = {26, 27};

// Top LED strip

#define TOPLED_NUM 13

#define TOPLED_PIN 27

CRGB leds2[TOPLED_NUM];

//LED ranges for light up

const int allie1Range2[] = {0, 2};

const int allie2Range2[] = {5, 7};

const int allie3Range2[] = {10, 12};

// Buttons and Keys

const int buttonPins[] = {30, 31, 32,

                         41, 40, 39, 38, 37, 36, 35};

const int audioLedPin = 24;

const int songPin = 28;

const int modePin = 29;

// logic variables

int number_of_keys = 3;

const int max_keys = 10;

int modeState = 0;

int lastModeState = HIGH;

elapsedMillis roundTimer;

bool pressedInTime = true;

bool wrongPress = false;

bool playing = false;

bool lastSongButtonState = HIGH;

unsigned long lastSongPressTime = 0;

unsigned long debounceDelay = 200;

int currentKey = -1;

int lastKey = -1;

// setup

//.............................................................................

void setup() {

 Serial.begin(9600);

 delay(1000);

 Serial.println("Starting Setup");

 // LED strips

 for (int i = 0; i < KEYLED_NUM; i++) {

   leds[i] = CRGB::Black;

 }

 for (int i = 0; i < TOPLED_NUM; i++) {

   leds2[i] = CRGB::Black;

 }

 FastLED.show();

 FastLED.addLeds<WS2812B, KEYLED_PIN, GRB>(leds, KEYLED_NUM);

 FastLED.addLeds<WS2812B, TOPLED_PIN, GRB>(leds2, TOPLED_NUM);

 // audio LED

 pinMode(audioLedPin, OUTPUT);

 digitalWrite(audioLedPin, LOW);

 // Keys

 for (int i = 0; i < max_keys; i++) {

   pinMode(buttonPins[i], INPUT_PULLUP);

 }

 // song button and mode switch

 pinMode(songPin, INPUT_PULLUP);

 pinMode(modePin, INPUT_PULLUP);

 // Audio

 AudioMemory(8);

 sgtl5000_1.enable();

 sgtl5000_1.volume(0.8);

 // SD card

 while (!(SD.begin(SDCARD_CS_PIN))) {

   Serial.println("SD card failed.");

   delay(1000);

 }

 Serial.println("Setup done");

}

// Function for playing music

// play and stop music

void playFile(const char *filename, bool playing)

{

 // if playing, play music

 // else, stop music

 if (playing) {

   playWav1.play(filename);

   Serial.print("playing music");

 }

 else {

   playWav1.stop();

   Serial.print("stopped music");

 }

 delay(25);

}

// Function for lighting LEDs

// lights correct LEDs

// Allie 1 - 0

// Allie 2 - 1

// Allie 3 - 2

//Player 1 - 3

//Player 2 - 4

//Player 3 - 5

//Player 4 - 6

//Player 5 - 7

//Player 6 - 8

//Player 7 - 9

void lightLEDs(int currentKey)

{

 // select the LEDs that need to be light up on the LED strip

 // according to the currentKey

 if (currentKey == 0) { //Allie 1

   for (int i = allie1Range[0]; i <= allie1Range[1]; i++) {

     leds[i] = CRGB(255, 0, 0); //red

   }

   for (int i = allie1Range2[0]; i <= allie1Range2[1]; i++) {

     leds2[i] = CRGB(255, 0, 0); //red

   }

 }

 else if (currentKey == 1) { //Allie 2

   for (int i = allie2Range[0]; i <= allie2Range[1]; i++) {

     leds[i] = CRGB(0, 255, 0); //green

   }

   for (int i = allie2Range2[0]; i <= allie2Range2[1]; i++) {

     leds2[i] = CRGB(0, 255, 0); //green

   }

 }

 else if (currentKey == 2) { //Allie 3

   for (int i = allie3Range[0]; i <= allie3Range[1]; i++) {

     leds[i] = CRGB(0, 0, 255); //blue

   }

   for (int i = allie3Range2[0]; i <= allie3Range2[1]; i++) {

     leds2[i] = CRGB(0, 0, 255); //blue

   }

 }

 else if (currentKey == 3) { //Player 1

   for (int i = pl1Range[0]; i <= pl1Range[1]; i++) {

     leds[i] = CRGB(255, 0, 0); //red

   }

 }

 else if (currentKey == 4) { //Player 2

   for (int i = pl2Range[0]; i <= pl2Range[1]; i++) {

     leds[i] = CRGB(255, 50, 0); //orange

   }

 }

 else if (currentKey == 5) { //Player 3

   for (int i = pl3Range[0]; i <= pl3Range[1]; i++) {

     leds[i] = CRGB(255, 225, 0); //yellow

   }

 }

 else if (currentKey == 6) { //Player 4

   for (int i = pl4Range[0]; i <= pl4Range[1]; i++) {

     leds[i] = CRGB(0, 255, 0); //green

   }

 }

 else if (currentKey == 7) { //Player 5

   for (int i = pl5Range[0]; i <= pl5Range[1]; i++) {

     leds[i] = CRGB(0, 164, 255); //blue

   }

 }

 else if (currentKey == 8) { //Player 6

   for (int i = pl6Range[0]; i <= pl6Range[1]; i++) {

     leds[i] = CRGB(125, 0, 255); //purple

   }

 }

 else if (currentKey == 9) { //Player 7

   for (int i = pl7Range[0]; i <= pl7Range[1]; i++) {

     leds[i] = CRGB(255, 0, 214); //pink

   }

 }

 FastLED.show();

}

// Function for changing song

void nextSong() {

 // increments the next song

 // has music continue playing if music was playing before

 playFile(songList[songIndex], false);

 songIndex = (songIndex + 1) % numberofsongs;

 Serial.print("now playing: ");

 Serial.println(songList[songIndex]);

 if (playing) {

   playFile(songList[songIndex], true);

 }

}

// loop

//.............................................................................

void loop() {

  // check if the song button has been pressed

 bool songReading = digitalRead(songPin);

 if (songReading == LOW

     && lastSongButtonState == HIGH

     && millis() - lastSongPressTime > debounceDelay) {

   nextSong();

   lastSongPressTime = millis();

 }

 lastSongButtonState = songReading;

 // check if the mode switch switches

 modeState = digitalRead(modePin);

 if (modeState != lastModeState) {

   if (modeState == HIGH) {

       number_of_keys = 3;

       Serial.println("Solo player");

   } else {

       number_of_keys = max_keys;

       Serial.println("Multi-player");

   }

   lastModeState = modeState;

   // reset game and LEDs

   lastKey = -1;

   currentKey = -1;

   pressedInTime = false;

   wrongPress = false;

   roundTimer = 0;

   for (int i = 0; i < KEYLED_NUM; i++) {

       leds[i] = CRGB::Black;

   }

   for (int i = 0; i < TOPLED_NUM; i++) {

       leds2[i] = CRGB::Black;

   }

   FastLED.show();

   return;

 }

 // pick a random key

 if (currentKey == -1) {

   do {

     currentKey = random(0, number_of_keys);

   } while (currentKey == lastKey);

   lastKey = currentKey;

 }

 // light LEDs according to picked LED

 for (int i = 0; i < KEYLED_NUM; i++) {

   leds[i] = CRGB::Black;

 }

 for (int i = 0; i < TOPLED_NUM; i++) {

   leds2[i] = CRGB::Black;

 }

 FastLED.show();

 lightLEDs(currentKey);

 // reset round

 roundTimer = 0;

 pressedInTime = false;

 wrongPress = false;

 // pick time constraint for the round

 float thisRoundLimit;

 if (currentKey >= 0 && currentKey <= 2) {

     thisRoundLimit = allieTime;

 } else {

     thisRoundLimit = playerTime;

 }

 // key pressing logic

 bool pressedFlag = false;

 while (roundTimer < (thisRoundLimit * 1000)

        && !pressedFlag) {

   // check if song button has been pressed

   bool reading = digitalRead(songPin);

   if (reading == LOW

       && lastSongButtonState == HIGH

       && millis() - lastSongPressTime > debounceDelay) {

     nextSong();

     lastSongPressTime = millis();

   }

   lastSongButtonState = reading;

   // check if button pressed is correct

   for (int i = 0; i < number_of_keys; i++) {

     if (digitalRead(buttonPins[i]) == LOW) {

       if (i == currentKey) {

         pressedInTime = true;

       } else {

         wrongPress = true;

       }

       delay(200);

       while (digitalRead(buttonPins[i]) == LOW) {

         delay(5);

       }

       pressedFlag = true;

       break;

     }

   }

   delay(5);

 }

 // start playing music and pick new LED or stop playing music

 if (pressedInTime && !wrongPress) { // correct press

   digitalWrite(audioLedPin, HIGH);

   if (!playing) {

     playFile(songList[songIndex], true);

     playing = true;

   }

   // pick next key

   do {

     currentKey = random(0, number_of_keys);

   } while (currentKey == lastKey);

   lastKey = currentKey;

 } else { // pressed incorrectly

   digitalWrite(audioLedPin, LOW);

   if (playing) {

     playFile(songList[songIndex], false);

     playing = false;

   }

 }

 delay(50);

}

assistivedevice - Downloadable file to the Arduino code for the Teensy 4.1

3dmodel.dwg - overall digital model of the physical design of the device

3mm-lasercut.dxf - laser cut file for physical box (3 mm)

6mm-lasercut.dxf - laser cut file for physical box (6 mm)

blackkeyL-3dprint.stl - 3D print of large black keys of piano

blackkeyS-3dprint.stl - 3D print of small black keys of piano

cover-3dprint.stl - 3D print of top lid of physical box

keys-lasercut.dxf - laser cut file for acrylic keys (6 mm)