Brandon Mao & Isaac Cheaz - Inmi Lee
Throughout brainstorming, we decided that we wanted to provide an experience that involved music through different interactive elements. We eventually decided to base the project around synesthesia because of its unfamiliarity in the public and my close connection with someone who has it.
Synesthesia is a biological phenomenon in which people have “sensory crossovers”, or when one sensory input elicits another involuntary sensory response. In the most commonly diagnosed form of synesthesia, chromesthesia, auditory signals cause a tinting of the vision according to pitch.
The design of the project is split into two sections. The first being the auditory responsive glasses, which I worked on, and the second being the number/light grid, which Isaac worked on.
The choice to construct glasses and a light grid are to exemplify the two most common variations of synesthesia: chromesthesia and grapheme-color synesthesia, respectively. The goal in separating the project in this way was to illustrate that synesthesia is not a black and white condition, with over 60 different observable variations. ¹
¹ Cleveland Clinic: https://my.clevelandclinic.org/health/symptoms/24995-synesthesia.
Through user testing and the final presentation, a common critique was how distinctly separate the two projects are. This is absolutely true, but while combining the two projects would be more visually appealing and cohesive, it would bombard the person interacting with it with way too many options on where to start. In an ideal setting, this would be adapted to a larger installation where people can go from one station to another, experiencing different forms of synesthesia, but for the sake of this project, it was left as two stations.
Essentially, there was no good way to combine the projects while still being straightforward enough. We learned from the midterm project that even with just 4 inputs, it was difficult to know the order of interaction, so limiting and separating things as much as possible was decided on.
Above: Nick, one of our user testers
Right: Original conception idea of combining both the numbers and objects into one board
Overall the fabrication process was not too difficult. We used a mixture of laser cutting, 3D printing, woodworking, and soldering to create the finished pieces.
The original idea of a physical piano failed because I measured stuff incorrectly when designing the the model which led to the buttons not fitting. Testing with buttons, I also couldn’t set up a way to have the note play only when it was being held down. I was using processing to play audio files as opposed to a frequency, so the repetitive inputs made the audio jumpy and not piano-sounding.
This led to the idea of objects, which were found on Thingiverse:
The glasses and lenses were also 3D printed, the frames being a modified version of: https://www.thingiverse.com/thing:21967
In order to get the glasses to actually tint the wearers vision, originally I experimented with rgb leds. These didn’t supply enough light and were way too hard to control and wire. I got a cut strip of neopixels which perfectly slotted into my frames. This, combined with the clear lenses we cut was enough to effectively tint the wearer’s vision. It works better in the dark, but at full brightness works in room lighting.
CODING playNote();
I wrote programs that would play the notes and do the color according to the colors my friend with synesthesia sees.
MAPPING AUDIO:
I then began to rig the chosen audio clips to the notes that they were playing. This was done using the millis() function and was mapped using audacity.
AUDIO USED:
Boat: https://www.youtube.com/watch?v=Zx6DDVGE6nA
Washing machine: https://www.youtube.com/watch?v=jZ6-wDhauB4
Freddy fazbear: https://www.youtube.com/watch?v=b-9n6GWHZQg
Big ben clock/ doorbell: https://www.youtube.com/watch?v=eqjFmsZGBSc
The wiring was rather simple and the box was created using Makercase. The pieces were glued together and all was running smoothly. The pieces of interaction were painted red to hint at what to do with them.
The sensors chosen for each item are as follows:
Boat: slide potentiometer (mimics boats moving)
Doorbell: button (like a regular doorbell)
Laundry Machine: magnet to detect if the door was open
Freddy: button (originally meant to be mounted in his nose to resemble the squeak of his nose from the game)
The construction of the number/color box was a mixture of soldering, laser cutting, and internal wiring. The box was made using Makercase and the neopixels were cut to provide enough length for the 10 different numbers. The neopixel strips all had to be soldered and connected to an arduino mega.
Originally we were going to illuminate the numbers through slits in the box, similar to a stencil, but went with holes instead. These offered more light.
The frame was built up with pieces of wood and fabric to give a sort of booth feel to the display and have the colors stand out more in a bright background.
The goal of this project was to introduce audiences to synesthesia through the interactive experiences of sound-responsive glasses and a color-association light grid. By recreating sensory pairings, the project enabled participants to explore how synesthetes perceive the world. While engaging and visually appealing, the experience often required verbal explanation to clarify the connections, indicating room for improvement in self-explanatory design.
Audience interactions showed curiosity and engagement, meeting the project’s goal of exploration and learning. However, clearer instructions and a more cohesive integration of the two components could have made the project better overall.
With more time, I would refine the glasses’ usability (easier to wear and the finnicky sensors), simplify the light grid’s interface, and create more different synesthetic experiences in a larger space. Challenges like technical limitations and communicating abstract concepts made it clear that there isn't ONE best solution to present a topic, and that offering your own creative take on it could be worthwhile.
Overall, the project successfully raised awareness about synesthesia and taught me to think less didactically when it comes to creative interactive art installations.
Final showcase:
SOURCES:
https://www.thesynesthesiatree.com/2021/04/the-most-common-types-of-synesthesia.html
https://www.disabled-world.com/disability/awareness/famous/famous-synesthesia.php#google_vignette
https://www.betterhelp.com/advice/synesthesia/the-many-types-of-synesthesia-explained/
https://study.com/academy/lesson/what-is-grapheme-color-synesthesia.html
Boat: https://www.youtube.com/watch?v=Zx6DDVGE6nA
Washing machine: https://www.youtube.com/watch?v=jZ6-wDhauB4
Freddy fazbear: https://www.youtube.com/watch?v=b-9n6GWHZQg
Big ben clock/ doorbell: https://www.youtube.com/watch?v=eqjFmsZGBSc
Boat: https://www.thingiverse.com/thing:763622
Washing Machine: https://www.thingiverse.com/thing:3351639
Freddy: https://www.thingiverse.com/thing:635885
Glasses: https://www.thingiverse.com/thing:21967
#include <FastLED.h>
#define NUM_LEDS 14 // How many leds on your strip?
#define DATA_PIN 3
CRGB leds[NUM_LEDS];
#define NUM_OF_VALUES_FROM_PROCESSING 1
int processing_values[NUM_OF_VALUES_FROM_PROCESSING];
int washing = 5;
int freddy = 6;
int door = 7;
int boat = A2;
long Start0;
long Start;
long Start2;
long Start3;
long Start4;
int calling = 1;
void setup() {
Serial.begin(9600);
FastLED.addLeds<NEOPIXEL, DATA_PIN>(leds, NUM_LEDS);
FastLED.setBrightness(100);
//magnetic sensor
pinMode(washing, INPUT);
//button
pinMode(freddy, INPUT);
//button
pinMode(door, INPUT);
//potentiometer
pinMode(boat, INPUT);
}
void loop() {
getSerialData();
if (calling == 1 && digitalRead(washing) == 0) {
Start = millis()+100;
calling = 0;
}
if (calling == 1 && digitalRead(freddy) == 1) {
Start2 = millis()+100;
calling = 0;
}
if (calling == 1 && digitalRead(door) == 1) {
Start3 = millis()+100;
calling = 0;
}
if (calling == 1 && analogRead(boat) >= 700) {
Start4 = millis()+100;
calling = 0;
}
//if (analogRead(A0) >= 800) {
//playSound(1);}
if (digitalRead(washing) == 0) {
playSound(2);
} else if (digitalRead(freddy) == 1) {
playSound(3);
} else if (digitalRead(door) == 1) {
playSound(4);
} else if (analogRead(A2) >= 700) {
playSound(5);
} else if (processing_values[0] == 1) {
playNote('C');
}
else if (processing_values[0] == 2) {
playNote('V');
}
else if (processing_values[0] == 3) {
playNote('D');
}
else if (processing_values[0] == 4) {
playNote('W');
}
else if (processing_values[0] == 5) {
playNote('E');
}
else if (processing_values[0] == 6) {
playNote('F');
}
else if (processing_values[0] == 7) {
playNote('X');
}
else if (processing_values[0] == 8) {
playNote('G');
}
else if (processing_values[0] == 9) {
playNote('Y');
}
else if (processing_values[0] == 10) {
playNote('A');
}
else if (processing_values[0] == 11) {
playNote('Z');
}
else if (processing_values[0] == 12) {
playNote('B');
} else {
playSound(6);
Serial.print("6");
Serial.println();
playNote('N');
calling = 1;
}
//Serial.print(calling);
//Serial.print(analogRead(A0));
delay(20);
}
void getSerialData() {
static int tempValue = 0; // the "static" makes the local variable retain its value between calls of this function
static int tempSign = 1;
static int valueIndex = 0;
while (Serial.available()) {
char c = Serial.read();
if (c >= '0' && c <= '9') {
// received a digit:
// multiply the current value by 10, and add the character (converted to a number) as the last digit
tempValue = tempValue * 10 + (c - '0');
} else if (c == '-') {
// received a minus sign:
// make a note to multiply the final value by -1
tempSign = -1;
} else if (c == ',' || c == '\n') {
// received a comma, or the newline character at the end of the line:
// update the processing_values array with the temporary value
if (valueIndex < NUM_OF_VALUES_FROM_PROCESSING) { // should always be the case, but double-check
processing_values[valueIndex] = tempValue * tempSign;
}
// get ready for the new data by resetting the temporary value and sign
tempValue = 0;
tempSign = 1;
if (c == ',') {
// move to dealing with the next entry in the processing_values array
valueIndex = valueIndex + 1;
} else {
// except when we reach the end of the line
// go back to the first entry in this case
valueIndex = 0;
}
}
}
}
void playSound(int type) {
//bird song
if (type == 1) {
Serial.print("1");
Serial.println();
}
//washing machine song
if (type == 2) {
Serial.print("2");
Serial.println();
if (millis() - Start < 659) {
playNote('V');
}
if (millis() - Start > 659 && millis() - Start < 855) {
playNote('X');
}
if (millis() - Start > 855 && millis() - Start < 1058) {
playNote('F');
}
if (millis() - Start > 1058 && millis() - Start < 1270) {
playNote('W');
}
if (millis() - Start > 1270 && millis() - Start < 1864) {
playNote('V');
}
if (millis() - Start > 1864 && millis() - Start < 2451) {
playNote('Z');
}
if (millis() - Start > 2451 && millis() - Start < 2661) {
playNote('B');
}
if (millis() - Start > 2661 && millis() - Start < 2862) {
playNote('V');
}
if (millis() - Start > 2862 && millis() - Start < 3045) {
playNote('W');
}
if (millis() - Start > 3045 && millis() - Start < 3249) {
playNote('Y');
}
if (millis() - Start > 3249 && millis() - Start < 3460) {
playNote('Z');
}
if (millis() - Start > 3460 && millis() - Start < 3656) {
playNote('B');
}
if (millis() - Start > 3656 && millis() - Start < 4250) {
playNote('Z');
}
if (millis() - Start > 4250 && millis() - Start < 4861) {
playNote('V');
}
if (millis() - Start > 4861 && millis() - Start < 5463) {
playNote('V');
}
if (millis() - Start > 5463 && millis() - Start < 5642) {
playNote('X');
}
if (millis() - Start > 5642 && millis() - Start < 5820) {
playNote('F');
}
if (millis() - Start > 5820 && millis() - Start < 6041) {
playNote('W');
}
if (millis() - Start > 6041 && millis() - Start < 6652) {
playNote('V');
}
if (millis() - Start > 6652 && millis() - Start < 7262) {
playNote('X');
}
if (millis() - Start > 7262 && millis() - Start < 7458) {
playNote('X');
}
if (millis() - Start > 7458 && millis() - Start < 7670) {
playNote('Y');
}
if (millis() - Start > 7670 && millis() - Start < 7857) {
playNote('X');
}
if (millis() - Start > 7857 && millis() - Start < 8060) {
playNote('F');
}
if (millis() - Start > 8060 && millis() - Start < 8280) {
playNote('W');
}
if (millis() - Start > 8280 && millis() - Start < 8451) {
playNote('F');
}
if (millis() - Start > 8451 && millis() - Start < 9575) {
playNote('F');
}
if (millis() - Start > 9575) {
playNote('N');
}
}
//freddy song
if (type == 3) {
Serial.print("3");
Serial.println();
if (millis() - Start2 > 631 && millis() - Start2 < 1191) {
playNote('C');
}
if (millis() - Start2 > 1191 && millis() - Start2 < 1688) {
playNote('D');
}
if (millis() - Start2 > 1688 && millis() - Start2 < 1929) {
playNote('C');
}
if (millis() - Start2 > 3368 && millis() - Start2 < 3637) {
playNote('A');
}
if (millis() - Start2 > 3637 && millis() - Start2 < 4098) {
playNote('G');
}
if (millis() - Start2 > 4098 && millis() - Start2 < 4272) {
playNote('A');
}
if (millis() - Start2 > 4272 && millis() - Start2 < 4598) {
playNote('Z');
}
if (millis() - Start2 > 4598 && millis() - Start2 < 5190) {
playNote('A');
}
if (millis() - Start2 > 5190 && millis() - Start2 < 5768) {
playNote('Z');
}
if (millis() - Start2 > 5768 && millis() - Start2 < 6346) {
playNote('G');
}
if (millis() - Start2 > 6346 && millis() - Start2 < 7434) {
playNote('C');
}
if (millis() - Start2 > 7434 && millis() - Start2 < 8016) {
playNote('A');
}
if (millis() - Start2 > 8016 && millis() - Start2 < 8502) {
playNote('F');
}
if (millis() - Start2 > 8502 && millis() - Start2 < 8732) {
playNote('D');
}
if (millis() - Start2 > 8732 && millis() - Start2 < 8909) {
playNote('G');
}
if (millis() - Start2 > 8909 && millis() - Start2 < 9200) {
playNote('C');
}
if (millis() - Start2 > 9200) {
playNote('N');
}
}
//door noises
if (type == 4) {
Serial.print("4");
Serial.println();
if (millis() - Start3 > 296 && millis() - Start3 < 1010) {
playNote('A');
}
if (millis() - Start3 > 1010 && millis() - Start3 < 1701) {
playNote('V');
}
if (millis() - Start3 > 1701 && millis() - Start3 < 2361) {
playNote('B');
}
if (millis() - Start3 > 2361 && millis() - Start3 < 3469) {
playNote('E');
}
if (millis() - Start3 > 3469 && millis() - Start3 < 4373) {
playNote('N');
}
if (millis() - Start3 > 4373 && millis() - Start3 < 5079) {
playNote('E');
}
if (millis() - Start3 > 5079 && millis() - Start3 < 5717) {
playNote('B');
}
if (millis() - Start3 > 5717 && millis() - Start3 < 6392) {
playNote('V');
}
if (millis() - Start3 > 6392 && millis() - Start3 < 7334) {
playNote('A');
}
if (millis() - Start3 > 7334 && millis() - Start3 < 7713) {
playNote('N');
}
if (millis() - Start3 > 7713 && millis() - Start3 < 9118) {
playNote('A');
}
if (millis() - Start3 > 9118) {
playNote('N');
}
}
//boat horn
if (type == 5) {
Serial.print("5");
Serial.println();
if (millis() - Start4 > 633 && millis() - Start4 < 3757) {
playNote('V');
}
if (millis() - Start4 > 3757) {
playNote('N');
}
}
if (type == 6) {
playNote('N');
}
}
void playNote(char note) {
if (note == 'C') {
for (int i = 0; i < NUM_LEDS; i++) {
leds[i] = CRGB(120, 6, 6);
FastLED.show();
}
}
if (note == 'V') {
for (int i = 0; i < NUM_LEDS; i++) {
leds[i] = CRGB(128, 0, 32);
FastLED.show();
}
}
if (note == 'D') {
for (int i = 0; i < NUM_LEDS; i++) {
leds[i] = CRGB(171, 108, 32);
FastLED.show();
}
}
if (note == 'W') {
for (int i = 0; i < NUM_LEDS; i++) {
leds[i] = CRGB(47, 217, 247);
FastLED.show();
}
}
if (note == 'E') {
for (int i = 0; i < NUM_LEDS; i++) {
leds[i] = CRGB(75, 240, 252);
FastLED.show();
}
}
if (note == 'F') {
for (int i = 0; i < NUM_LEDS; i++) {
leds[i] = CRGB(109, 36, 255);
FastLED.show();
}
}
if (note == 'X') {
for (int i = 0; i < NUM_LEDS; i++) {
leds[i] = CRGB(76, 59, 170);
FastLED.show();
}
}
if (note == 'G') {
for (int i = 0; i < NUM_LEDS; i++) {
leds[i] = CRGB(255, 229, 180);
FastLED.show();
}
}
if (note == 'Y') {
for (int i = 0; i < NUM_LEDS; i++) {
leds[i] = CRGB(219, 242, 5);
FastLED.show();
}
}
if (note == 'A') {
for (int i = 0; i < NUM_LEDS; i++) {
leds[i] = CRGB(255, 238, 0);
FastLED.show();
}
}
if (note == 'Z') {
for (int i = 0; i < NUM_LEDS; i++) {
leds[i] = CRGB(101, 139, 56);
FastLED.show();
}
}
if (note == 'B') {
for (int i = 0; i < NUM_LEDS; i++) {
leds[i] = CRGB(0, 255, 0);
FastLED.show();
}
}
if (note == 'N') {
for (int i = 0; i < NUM_LEDS; i++) {
leds[i] = CRGB(0, 0, 0);
FastLED.show();
}
}
}
import processing.serial.*;
import processing.sound.*;
Serial serialPort;
SoundFile bird;
SoundFile washing;
SoundFile freddy;
SoundFile door;
SoundFile boat;
SoundFile C;
SoundFile V;
SoundFile D;
SoundFile W;
SoundFile E;
SoundFile F;
SoundFile X;
SoundFile G;
SoundFile Y;
SoundFile A;
SoundFile Z;
SoundFile B;
int calling = 1;
int NUM_OF_VALUES_FROM_ARDUINO = 1;
int NUM_OF_VALUES_FROM_PROCESSING = 1;
int arduino_values[] = new int[NUM_OF_VALUES_FROM_ARDUINO];
int processing_values[] = new int[NUM_OF_VALUES_FROM_PROCESSING];
void setup() {
size(1000, 600);
background(100);
noStroke();
rectMode(CENTER);
textSize(30);
serialPort = new Serial(this, "COM6", 9600);
bird = new SoundFile(this, "bird.mp3");
washing = new SoundFile(this, "washing.mp3");
freddy = new SoundFile(this, "freddy.mp3");
door = new SoundFile(this, "door.mp3");
boat = new SoundFile(this, "boat.mp3");
C = new SoundFile(this, "C.mp3");
V = new SoundFile(this, "V.mp3");
D = new SoundFile(this, "D.mp3");
W = new SoundFile(this, "W.mp3");
E = new SoundFile(this, "E.mp3");
F = new SoundFile(this, "F.mp3");
X = new SoundFile(this, "X.mp3");
G = new SoundFile(this, "G.mp3");
Y = new SoundFile(this, "Y.mp3");
A = new SoundFile(this, "A.mp3");
Z = new SoundFile(this, "Z.mp3");
B = new SoundFile(this, "B.mp3");
}
int n = 260;
int m = 25;
int j = 30;
void draw() {
for (int i=300; i<750; i+=70) {
drawKey(1, i, 0);
}
drawKey(0, 75+n, 0);
drawKey(0, 145+n, 10);
drawKey(0, 285+n, 20);
drawKey(0, 355+n, 30);
drawKey(0, 425+n, 20);
fill(255, 0, 0);
rect(width/2, height-70, 150, 60);
circle(width/2-75, height-70, 60);
circle(width/2+75, height-70, 60);
fill(255);
text("stop", width/2-30, height-65);
fill(0);
getSerialData();
int type = arduino_values[0];
if (type == 1 && bird.isPlaying() == false) {
stopSound();
bird.play();
}
if (type == 2 && washing.isPlaying() == false && calling == 1) {
stopSound();
washing.play();
calling = 0;
processing_values[0] = 0;
}
if (type == 3 && freddy.isPlaying() == false && calling == 1) {
stopSound();
freddy.play();
calling = 0;
processing_values[0] = 0;
}
if (type == 4 && door.isPlaying() == false && calling == 1) {
stopSound();
door.play();
calling = 0;
processing_values[0] = 0;
}
if (type == 5 && boat.isPlaying() == false && calling == 1) {
stopSound();
boat.play();
calling = 0;
processing_values[0] = 0;
}
if (type ==6) {
bird.stop();
washing.stop();
freddy.stop();
door.stop();
boat.stop();
calling = 1;
}
sendSerialData();
}
// the helper function below receives the values from Arduino
// in the "arduino_values" array from a connected Arduino
// running the "serial_AtoP_arduino" sketch
// (You won't need to change this code.)
void getSerialData() {
while (serialPort.available() > 0) {
String in = serialPort.readStringUntil( 10 ); // 10 = '\n' Linefeed in ASCII
if (in != null) {
print("From Arduino: " + in);
String[] serialInArray = split(trim(in), ",");
if (serialInArray.length == NUM_OF_VALUES_FROM_ARDUINO) {
for (int i=0; i<serialInArray.length; i++) {
arduino_values[i] = int(serialInArray[i]);
}
}
}
}
}
void drawKey(int c, int x, int y) {
if (c ==1) {
fill(250);
rect(x, height/2, 55, height);
//circle(x, height/2+75, 60);
//circle(x, height/2-75, 60);
}
if (c ==0) {
fill(10);
rect(x, height/2-40-y, 50, 100);
circle(x, height/2+10-y, 50);
circle(x, height/2-90-y, 50);
}
}
void mousePressed() {
if (mouseY > height/2-90 && mouseY< height/2 +10) {
if (mouseX> 335-m && mouseX< 385-m) {
background(30);
stopSound();
V.play();
processing_values[0] = 2;
}
if (mouseX> 405-m && mouseX< 455-m) {
background(70);
stopSound();
W.play();
processing_values[0] = 4;
}
if (mouseX> 545-m && mouseX< 595-m) {
background(130);
stopSound();
X.play();
processing_values[0] = 7;
}
if (mouseX> 615-m && mouseX< 665-m) {
background(170);
stopSound();
Y.play();
processing_values[0] = 9;
}
if (mouseX> 685-m && mouseX< 735-m) {
background(210);
stopSound();
Z.play();
processing_values[0] = 11;
}
} //else if (mouseY> height/2-75 && mouseY < height/2+75) {
else {
if (mouseX> 300-j && mouseX< 370-j) {
background(20);
stopSound();
C.play();
processing_values[0] = 1;
}
if (mouseX> 370-j && mouseX< 440-j) {
background(50);
stopSound();
D.play();
processing_values[0] = 3;
}
if (mouseX> 440-j && mouseX< 510-j) {
background(90);
stopSound();
E.play();
processing_values[0] = 5;
}
if (mouseX> 510-j && mouseX< 580-j) {
background(110);
stopSound();
F.play();
processing_values[0] = 6;
}
if (mouseX> 580-j && mouseX< 650-j) {
background(150);
stopSound();
G.play();
processing_values[0] = 8;
}
if (mouseX> 650-j && mouseX< 720-j) {
background(190);
stopSound();
A.play();
processing_values[0] = 10;
}
if (mouseX> 720-j && mouseX< 790-j) {
background(220);
stopSound();
B.play();
processing_values[0] = 12;
}
}
if (mouseY > height-100) {
stopSound();
processing_values[0] = 0;
background(211, 149, 232);
}
}
void stopSound() {
C.stop();
D.stop();
E.stop();
F.stop();
G.stop();
A.stop();
B.stop();
V.stop();
W.stop();
X.stop();
Y.stop();
Z.stop();
}
void sendSerialData() {
String data = "";
for (int i=0; i<processing_values.length; i++) {
data += processing_values[i];
// if i is less than the index number of the last element in the values array
if (i < processing_values.length-1) {
data += ","; // add splitter character "," between each values element
}
// if it is the last element in the values array
else {
data += "\n"; // add the end of data character "n"
}
}
// write to Arduino
serialPort.write(data);
print("To Arduino: " + data); // this prints to the console the values going to Arduino
}
hi
it looks like you made it to the bottom of my webpage
published: December 12, 2024