Embedded Files
December 16th, 2024 by Isaac Cheaz
Prof: Inmi Lee
video.mp4Concept:
My project surrounded the concept of synesthesia. Notably, It was a type of synesthesia that dealt with numerals and an individual's association of them with colors. Seen at the top of my project: "Grapheme-Colour", was the type of the synesthesia I worked with. The concept of synesthesia was proposed by my partner Brandon. Back at home, he had somebody who had this medical condition and would see colors depending on the note he heard. Although the friend suffered from a different type of synesthesia, it still laid out the foundation of what my project would be about.
A More In Depth Sketch
Original Sketch Of Our Project's
The brainstorming part of this project might be the worst and most mundane section of the final. Before the concept of synesthesia, the project was going to be a musical garden in which users would be able to learn music theory when interacting with the 4 sections of the garden. The idea was to make an environment educational, interactive, artsy and be used by all ages. After focusing on the topic of synesthesia, I wanted to incorporate these 4 values into my project. However, looking back, I think the educational part was a bit looked over.
There were many ideas, with many of them including music and lights to create a spectacle for the users. The ideas had a strong leniency towards the connection between emotions and sensory inputs.
For this project, I honestly focused more on the output aspect rather than the input aspect. Since the inputs were going to be in the laptop, there was neglect towards the interactivity of the project.
I knew I wanted to work with Neopixels, one of the most recent components introduced in the recitations. I wanted someone that would stand out and would be relevant to the theme of Grapheme Colour Synesthesia.
Feedback From User-Testing
User Testing:
During user testing, there were strong comments given by one of the fellows about the unity our projects. For this project, I worked in a pair and was supposed create one product. However, since the topic of synesthesia can be very diverse, our project basically became two projects that tackled different types of synesthesia. Another comment I vaguely remember was the keyboard and mouse interaction. Andy, the head of interaction lab, pointed out my use of keyboard and mouse for the users to input the colors and recommended other components such as a potentiometer to control the RGB.
Fabrication and Production:
The most fun but also the most "rollercoaster" part of the project.
First time woodworking
Laptop breaking and having to improvise to fix it
Adding "hello" in different languages to give the top a design
Originally, I was going to make a square box that instead of the rectangular one seen in the final product. I even designed the box myself in cuttle. However, the size of the box didn't fit the laser cutter in the fabrication lab and also the nudges were the wrong size.
After cutting it and realizing my design was flawed. I asked 大琳 for help, he used a website to generate the dimensions and design of the box. The box was half as wide and had the square holes I designed for my previous (failed) box.
The components were simple: Neopixels, Arduino (Mega), and a TON of cables. Since synesthesia worked with colors and symbols. 10% were the components, 40% was the design and 50% was the coding.
The rest of the neopixels were too hard to put, but there were 10 more strips hooked to pins 4-13.
Before building the data visualization of my project, I only worked with two strips of Neopixels, and this was enough for the Arduino's 5V to handle. However, after adding 10 more strips, I had to hook up an external power source to the breadboard. Getting 10 more strips of Neopixels was also hard, since I requested them during the last weeks of interaction lab, they only had 6 strips left. Which left me no choice but to cut 5 in half, giving me a total of 10-half Neopixels.
Soldering them was a nightmare...
Coding:
The part of the final that gave me late nights at the IMA studio. My idea was to have serial communication from Processing to Arduino. Processing would intake the RGB codes picked out by the user and Arduino would receive the information via serial port and display the colors onto the neopixels. Sounds simple enough on paper... right?
173 lines of Arduino code
221 lines of Processing code
To be fair, the codes have a lot of redundancies and could be cleaned up to be around 120 lines at most.
Where I had the most problems was Arduino correctly displaying the colors on the neopixels. When I restarted the color survey, it would display only 90% of the rgb codes.
Interface:
There were also a lot of changes in the interface of the survey. Originally, I was going to make a simple color picker that displayed the selected color onto a square and a prompt that changed number after you confirmed your color. After many changes and taking inspiration from the top design, I added multiple languages onto the survey to accommodate more demographics and also made the survey easier to imply what it wants you to do.
I only showed the two most commonly selected languages
Conclusions:
My project was a success, at least when I was present around it. The concept of synesthesia wasn't exactly widely known around the world. I've noticed that people from west tend to recognize the term when they are in the field of IMA or arts. I've also found that Chinese citizens are more likely to know it, surprisingly at a young age too. I remember the many times I tried explaining it in English to be faced with confused looks and say 通感 and here "aaaaahhhhh".
My project was already pretty open to audiences of different background. This combined with the different languages incorporated into the survey made it even more accessible to everybody. The IMA show was honestly one of the happiest days I've had in this semester. I was so happy to see my project in action and being interacted by people outside the campus. My audience's interaction with my project was almost ideal. When I was around the project, I would explain the project and how the color picker worked and after 30 uses, it would display a colorful rainbow of people's association with colors with numbers.
30 people's colors from numbers 0-9
Improvements
This leads me to my room for improvements. If I had more time to do some tweaks, I would change the interface to make it look more sophisticated and easier to imply the uses. I noticed that many users weren't sure on how to use the color picker and would click confirm, creating a completely white palette of colors. I would also implement more sound effects since the IMA show's room was quieter than I expected.
Extras:
"Daily work at NYUSH's IMA"
"Picture 1: What is the first color that comes to your mind when you look at each number? My answer is of course the colors of the Shanghai Metro's lines."
A user posted my project onto their 小红书 (REDnote)
@nyushanghai featuring my project on their stories
Disassembly
Arduino
Arduino
#include <FastLED.h>
#define NUM_STRIPS 11
#define NUM_LEDS 30
#define DATA_PIN 3
#define NUM_OF_VALUES_FROM_PROCESSING 3
int dataPins[NUM_STRIPS] = { 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 };
CRGB leds[NUM_STRIPS][NUM_LEDS];
int rgbValues[NUM_STRIPS][NUM_LEDS][3];
int processing_values[NUM_OF_VALUES_FROM_PROCESSING];
int rgbCounter = 0;
int ledIndex[NUM_STRIPS] = { 0 };
bool waitingForNewData = false;
bool handshakeReceived = false;
int level = 0;
void setup() {
Serial.begin(100000);
delay(1000);
Serial.println("READY");
FastLED.addLeds<NEOPIXEL, 3>(leds[0], NUM_LEDS);
FastLED.addLeds<NEOPIXEL, 4>(leds[1], NUM_LEDS);
FastLED.addLeds<NEOPIXEL, 5>(leds[2], NUM_LEDS);
FastLED.addLeds<NEOPIXEL, 6>(leds[3], NUM_LEDS);
FastLED.addLeds<NEOPIXEL, 7>(leds[4], NUM_LEDS);
FastLED.addLeds<NEOPIXEL, 8>(leds[5], NUM_LEDS);
FastLED.addLeds<NEOPIXEL, 9>(leds[6], NUM_LEDS);
FastLED.addLeds<NEOPIXEL, 10>(leds[7], NUM_LEDS);
FastLED.addLeds<NEOPIXEL, 11>(leds[8], NUM_LEDS);
FastLED.addLeds<NEOPIXEL, 12>(leds[9], NUM_LEDS);
FastLED.addLeds<NEOPIXEL, 13>(leds[10], NUM_LEDS);
FastLED.setBrightness(100);
FastLED.clear();
FastLED.show();
delay(1000);
}
void loop() {
if (!handshakeReceived) {
if (getHandshake()) {
handshakeReceived = true;
Serial.println("HANDSHAKE RECEIVED");
}
return;
}
if (waitingForNewData) {
if (getHandshake()) {
waitingForNewData = false;
level++;
if (level>29){
level = 0;
}
rgbCounter = 0;
ledIndex[0] = 0;
for (int i = 0; i < NUM_LEDS; i++) {
leds[0][i] = CRGB::Black;
}
FastLED.show();
delay(100);
Serial.println("READY FOR NEW DATA");
}
return;
}
if (getSerialData()) {
int r = processing_values[0];
int g = processing_values[1];
int b = processing_values[2];
if (rgbCounter < 10 && ledIndex[0] < NUM_LEDS) {
for (int i = 0; i < 3; i++) {
if (ledIndex[0] + i < NUM_LEDS) {
leds[0][ledIndex[0] + i] = CRGB(r, g, b);
}
}
if (rgbCounter == 0) {
leds[1][level] = CRGB(r, g, b);
}
if (rgbCounter == 1) {
leds[2][level] = CRGB(r, g, b);
}
if (rgbCounter == 2) {
leds[3][level] = CRGB(r, g, b);
}
if (rgbCounter == 3) {
leds[4][level] = CRGB(r, g, b);
}
if (rgbCounter == 4) {
leds[5][level] = CRGB(r, g, b);
}
if (rgbCounter == 5) {
leds[6][level] = CRGB(r, g, b);
}
if (rgbCounter == 6) {
leds[7][level] = CRGB(r, g, b);
}
if (rgbCounter == 7) {
leds[8][level] = CRGB(r, g, b);
}
if (rgbCounter == 8) {
leds[9][level] = CRGB(r, g, b);
}
if (rgbCounter == 9) {
leds[10][level] = CRGB(r, g, b);
}
ledIndex[0] += 3;
}
rgbCounter++;
if (rgbCounter >= 10) {
waitingForNewData = true;
}
FastLED.show();
}
}
bool getHandshake() {
while (Serial.available()) {
String message = Serial.readStringUntil('\n');
if (message == "START") {
return true;
}
}
return false;
}
bool getSerialData() {
static int tempValue = 0;
static int valueIndex = 0;
while (Serial.available()) {
char c = Serial.read();
if (c >= '0' && c <= '9') {
tempValue = tempValue * 10 + (c - '0');
} else if (c == ',' || c == '\n') {
if (valueIndex < NUM_OF_VALUES_FROM_PROCESSING) {
processing_values[valueIndex] = tempValue;
valueIndex++;
}
tempValue = 0;
if (c == '\n') {
valueIndex = 0;
return true;
}
}
}
return false;
}
Processing
Processing
import processing.serial.*;
import processing.sound.*;
Serial serialPort;
SoundFile soundPlayer;
SoundFile coinSoundPlayer;
SoundFile levelSoundPlayer;
SoundFile coin2SoundPlayer;
SoundFile coin3SoundPlayer;
SoundFile coin4SoundPlayer;
SoundFile[] sounds;
color selectedColor = color(255);
int pickerWidth = 300;
int pickerHeight = 300;
int sliderY = 320;
float hueValue = 0;
float brightness = 255;
int num = 0;
int level = 0;
// Language selection variables
boolean languageSelected = false;
String selectedLanguage = "";
String[] languages = {"English", "Spanish", "Chinese", "Korean", "Japanese","Arabic", "Mongolian"};
int languageChoice = -1; // -1 means no choice yet
// Array to store user-selected RGB values
int[][] savedRGBs = new int[10][3]; // 10 numbers, each with an R, G, and B value
PImage english, spanish, chinese, korean, japanese, arabic, mongolian;
void setup() {
size(1280, 800);
frameRate(30);
colorMode(HSB, 255);
printArray(Serial.list());
selectedColor = color(255);
serialPort = new Serial(this, "COM7", 100000); // Adjust COM port as needed
// Load the MP3 files
soundPlayer = new SoundFile(this, "sound.mp3");
coinSoundPlayer = new SoundFile(this, "coin.mp3");
levelSoundPlayer = new SoundFile(this, "level.mp3");
coin2SoundPlayer = new SoundFile(this, "coin2.mp3");
coin3SoundPlayer = new SoundFile(this, "coin3.mp3");
coin4SoundPlayer = new SoundFile(this, "coin4.mp3");
sounds = new SoundFile[] {coinSoundPlayer, coin2SoundPlayer, coin3SoundPlayer, coin4SoundPlayer};
english = loadImage("english.jpg");
spanish = loadImage("spanish.jpg");
chinese = loadImage("chinese.jpg");
korean = loadImage("korean.jpg");
japanese = loadImage("japanese.jpg");
arabic = loadImage("arabic.jpg");
mongolian = loadImage("mongolian.jpg");
}
void draw() {
background(0);
if (!languageSelected) {
displayLanguageSelection();
return;
}
for (int x = 0; x < pickerWidth; x++) {
for (int y = 0; y < pickerHeight; y++) {
float saturation = map(x, 0, pickerWidth, 0, 255);
float darkValue = map(y, 0, pickerHeight, 255, 0);
stroke(hueValue, saturation, darkValue);
point(x, y);
}
}
for (int x = 0; x < pickerWidth; x++) {
float sliderHue = map(x, 0, pickerWidth, 0, 255);
stroke(sliderHue, 255, 255);
line(x, sliderY, x, sliderY + 20);
}
noStroke();
fill(0);
textSize(12);
text("Selected Color", 10, pickerHeight + 170);
fill(#808080);
rect(150/2, 350, 150, 100, 10);
textSize(30);
fill(255);
text("Confirm", 150, 400);
textSize(30);
fill(255);
//text(question, 750, 100);
if (selectedLanguage == "English") {
image(english, 400, 50, 750, 100);
} else if (selectedLanguage == "Spanish") {
image(spanish, 400, 50, 750, 100);
} else if (selectedLanguage == "Chinese") {
image(chinese, 400, 50, 750, 100);
} else if (selectedLanguage == "Korean") {
image(korean, 400, 50, 750, 100);
} else if (selectedLanguage == "Japanese") {
image(japanese, 400, 50, 750, 100);
}
else if (selectedLanguage == "Arabic") {
image(arabic, 400, 50, 750, 100);
}
else if (selectedLanguage == "Mongolian") {
image(mongolian, 400, 50, 750, 100);
}
fill(selectedColor);
textSize(300);
text(num, 750, 400);
}
void mousePressed() {
if (!languageSelected) {
for (int i = 0; i < languages.length; i++) {
if (mouseX > 490 && mouseX < 790 && mouseY > 105 + (i * 60) && mouseY < 155 + (i * 60)) {
selectedLanguage = languages[i];
int randomIndex = int(random(sounds.length));
sounds[randomIndex].play();
languageChoice = i;
languageSelected = true;
println("Language selected: " + selectedLanguage);
break;
}
}
return;
}
if (mouseX >= 0 && mouseX < pickerWidth && mouseY >= 0 && mouseY < pickerHeight && languageSelected == true) {
float saturation = map(mouseX, 0, pickerWidth, 0, 255);
float darkValue = map(mouseY, 0, pickerHeight, 255, 0);
selectedColor = color(hueValue, saturation, darkValue);
}
if (mouseX >= 0 && mouseX < pickerWidth && mouseY >= sliderY && mouseY <= sliderY + 20 && languageSelected == true) {
hueValue = map(mouseX, 0, pickerWidth, 0, 255);
}
if (mouseX >= 150/2 && mouseX <= 225 && mouseY >= 350 && mouseY <= 450 && languageSelected == true) {
int randomIndex = int(random(sounds.length));
sounds[randomIndex].play();
if (num < 10) {
storeRGB(num, selectedColor);
num++;
selectedColor = color(255);
}
if (num == 10) {
sendAllRGBData();
resetQuiz();
}
}
}
void displayLanguageSelection() {
fill(255);
textSize(40);
textAlign(CENTER, CENTER);
text("Select your language", width / 2, 50);
for (int i = 0; i < languages.length; i++) {
fill(200);
rect(490, 105 + (i * 60), 300, 50, 10);
fill(255);
textSize(30);
text(languages[i], width / 2, 130 + (i * 60));
}
}
void storeRGB(int index, color selectedColor) {
savedRGBs[index][0] = (int) red(selectedColor);
savedRGBs[index][1] = (int) green(selectedColor);
savedRGBs[index][2] = (int) blue(selectedColor);
}
ArrayList<int[]> allRGBHistory = new ArrayList<int[]>();
void sendAllRGBData() {
if (!soundPlayer.isPlaying()) {
delay(500);
soundPlayer.play();
}
serialPort.write("START\n"); // Always send START handshake
delay(100); // Small delay to ensure Arduino is ready
for (int i = 0; i < 10; i++) {
int r = savedRGBs[i][0];
int g = savedRGBs[i][1];
int b = savedRGBs[i][2];
serialPort.write(r + "," + g + "," + b + "\n");
println("Sent RGB: " + r + ", " + g + ", " + b);
delay(10); // Delay to ensure Arduino processes data
// Append the sent RGB values to the history array
int[] rgb = {r, g, b};
allRGBHistory.add(rgb);
}
languageSelected = false;
level++;
if (level >30){
delay(200);
levelSoundPlayer.play();
level = 0;
}
}
void resetQuiz() {
num = 0;
hueValue = 0;
selectedColor = color(255);
}
Page updated
Report abuse