You Are Art
You Are Art
May 18th, 2025 by Isaac Cheaz
Prof: J.H. Moon
Design and Composition
For Project B, I wanted to take some inspirations from the concept of interaction I learned in IXLab. I was sure that I wanted the project to be personalized and have different results for every user. But how can I do that?
I took heavy inspiration from group-generated art. An art piece that could be conjured up by many individuals, a way to denote a person's thoughts, ideas and creativity.
The picture on the left is from a book called "We Feel Fine" by Jonathan Harris. The project collects peopleβs emotions from across the internet and turns them into an interactive experience.Β
I loved the aesthetic of these small dots of colors that all form paths that diverge from what seems to resemble a heart, bringing back memories of Project's A creature.Β Β
After the introduction of ml5.js by Professor Moon, I knew that I wanted to mix these two concepts. The idea of mouse and keyboard interactions were mundane to me, having heard this comment from Andy last semester. This led me to playing around with the models and even using multiple ones, which made the program really lag...
For the prototype, I was trying to come up with a way to emphasize the "You" in "You are art", playing around with the face mesh and making it you part of the art piece.
Still, in addition to face mesh, I also wanted to explore my first idea: interactions without mouse and keyboard. I ended mixing FaceMesh and HandPose, with interactions based on the hand gestures you make with your hands.
Sadly, on a meeting with Professor Moon, he was straightforward with me, stating that there was no extra "twist" to this program that was unique to me. This left me pretty heartbroken honestly, but I knew what he said was true. I spent the next few days brainstorming and thinking of what I can add to this project that was "me".
I removed the FaceMesh, as suggested by Professor Moon and focused on the hand interactions.Β
On the User-Testing Day, a friend also mentioned that the lines reminded them of a notebook and being able to doodle on it would've been cool.
Honestly, I took a lot of advice that day, which made my project's trajectory turn 89 degrees. I knew I wanted to keep the line deformation to keep the theme of "You Are Art" while also balancing the amount lines to keep it from being too laggy.
I added an Intro to the Project, similar to when you open a new macbook. I wanted to give the audience that feeling, that the experience was going to be high quality. Also circling back to the inspirations, I also wanted to add the colorful circles to the project. Although they don't represent a single person's emotions like in "We Feel Fine", I feel the group of dots serve as one or two people's message (or doodle) would also work.
In addition, taking inspiration of what is interaction. The project also delved into question of what people would perceive an "OK" sign to be. Since the code only compared distances of keypoints on the fingers, it was also interesting to see what multiple hand gestures can achieve the same thing. I liked the idea of implied interaction, letting the audience discover the controls themselves instead of concrete instructions. This freedom added to the minimalistic aesthetic of the project and also produced very funny reactions when they discovered an odd hand gesture.
Technical
The coding was also an adventure.Β
let handPose;
let video;
let hands = [];
let options = { maxHands: 2 };
let showIntro = true;
let showVideo = false;
let NowStartDrawing = false;
let vidWidth = 1056;
let vidHeight = 792;
let drawingPoints = [];
let currentColor = 0;
function preload() {
Β Β handPose = ml5.handPose(options);
}
function setup() {
Β Β createCanvas(windowWidth, windowHeight);
Β Β colorMode(HSB, 255);
Β Β video = createCapture(VIDEO);
Β Β video.size(vidWidth, vidHeight);
Β Β video.hide();
Β Β handPose.detectStart(video, gotHands);
Β Β background(255);
}
function draw() {
Β Β background(255);
Β Β let elapsed = millis();
Β Β if (showIntro) {
Β Β Β Β let alpha;
Β Β Β Β if (elapsed < 2000) {
Β Β Β Β Β Β alpha = map(elapsed, 0, 2000, 0, 255);
Β Β Β Β } else if (elapsed < 7000) {
Β Β Β Β Β Β alpha = 255;
Β Β Β Β } else if (elapsed < 10000) {
Β Β Β Β Β Β alpha = map(elapsed, 7000, 10000, 255, 0);
Β Β Β Β } else if (elapsed >= 10000) {
Β Β Β Β Β Β showIntro = false;
Β Β Β Β Β Β showVideo = true;
Β Β Β Β Β Β return;
Β Β Β Β }
Β Β Β Β fill(0, alpha);
Β Β Β Β noStroke();
Β Β Β Β textAlign(CENTER, CENTER);
Β Β Β Β textSize(64);
Β Β Β Β text("YOU ARE ART", width / 2, height / 2);
Β Β }
Β Β let gridSize = 40;
Β Β if (frameCount % 3 == 0) {
Β Β Β Β video.loadPixels();
Β Β }
Β Β if (showVideo) {
Β Β Β Β let videoalpha = 0;
Β Β Β Β fill(0);
Β Β Β Β noStroke();
Β Β Β Β textAlign(LEFT, TOP);
Β Β Β Β textSize(16);
Β Β Β Β text("Make an OK sign to draw and play around to find the delete hand gesture!", width - vidWidth, 50);
Β Β Β Β push();
Β Β Β Β stroke(0);
Β Β Β Β strokeWeight(2);
Β Β Β Β line((width - vidWidth) / 2, 0, (width - vidWidth) / 2, height);
Β Β Β Β line(width - (width - vidWidth) / 2, 0, width - (width - vidWidth) / 2, height);
Β Β Β Β pop();
Β Β Β Β if (elapsed < 15000) {
Β Β Β Β Β Β videoalpha = map(elapsed, 10000, 15000, 0, 255);
Β Β Β Β } else {
Β Β Β Β Β Β videoalpha = 255;
Β Β Β Β Β Β NowStartDrawing = true;
Β Β Β Β }
Β Β Β Β push();
Β Β Β Β translate(width / 2, height / 2);
Β Β Β Β scale(-1, 1);
Β Β Β Β for (let y = 0; y < video.height; y += gridSize) {
Β Β Β Β Β Β stroke(0, videoalpha);
Β Β Β Β Β Β noFill();
Β Β Β Β Β Β beginShape();
Β Β Β Β Β Β for (let x = 0; x < video.width; x += gridSize) {
Β Β Β Β Β Β Β Β let index = (x + y * video.width) * 4;
Β Β Β Β Β Β Β Β let r = video.pixels[index + 0];
Β Β Β Β Β Β Β Β let g = video.pixels[index + 1];
Β Β Β Β Β Β Β Β let b = video.pixels[index + 2];
Β Β Β Β Β Β Β Β let avg = (r + g + b) / 3;
Β Β Β Β Β Β Β Β let yAdj = map(avg, 0, 255, 0, gridSize * 2);
Β Β Β Β Β Β Β Β curveVertex(x - vidWidth / 2, (y - vidHeight / 2) + yAdj);
Β Β Β Β Β Β }
Β Β Β Β Β Β endShape();
Β Β Β Β }
Β Β Β Β const connections = [
Β Β Β Β Β Β [0, 1], [1, 2], [2, 3], [3, 4],
Β Β Β Β Β Β [5, 6], [6, 7], [7, 8],
Β Β Β Β Β Β [9, 10], [10, 11], [11, 12],
Β Β Β Β Β Β [13, 14], [14, 15], [15, 16],
Β Β Β Β Β Β [0, 17], [17, 18], [18, 19], [19, 20],
Β Β Β Β Β Β [1, 5], [5, 9], [9, 13], [13, 17],
Β Β Β Β ];
Β Β Β Β for (let i = 0; i < hands.length; i++) {
Β Β Β Β Β Β let hand = hands[i];
Β Β Β Β Β Β let keypoints = hand.keypoints;
Β Β Β Β Β Β stroke(0);
Β Β Β Β Β Β strokeWeight(2);
Β Β Β Β Β Β for (let k = 0; k < connections.length; k++) {
Β Β Β Β Β Β Β Β let [i1, i2] = connections[k];
Β Β Β Β Β Β Β Β let kp1 = keypoints[i1];
Β Β Β Β Β Β Β Β let kp2 = keypoints[i2];
Β Β Β Β Β Β Β Β line(kp1.x - vidWidth / 2, kp1.y - vidHeight / 2, kp2.x - vidWidth / 2, kp2.y - vidHeight / 2);
Β Β Β Β Β Β }
Β Β Β Β }
Β Β Β Β if (NowStartDrawing) {
Β Β Β Β Β Β for (let i = 0; i < hands.length; i++) {
Β Β Β Β Β Β Β Β let hand = hands[i];
Β Β Β Β Β Β Β Β let keypoints = hand.keypoints;
Β Β Β Β Β Β Β Β let thumb = keypoints[4];
Β Β Β Β Β Β Β Β let index = keypoints[8];
Β Β Β Β Β Β Β Β let middle = keypoints[12];
Β Β Β Β Β Β Β Β let ring = keypoints[16];
Β Β Β Β Β Β Β Β let pinky = keypoints[20];
Β Β Β Β Β Β Β Β let centerX = (index.x + thumb.x) / 2;
Β Β Β Β Β Β Β Β let centerY = (index.y + thumb.y) / 2;
Β Β Β Β Β Β Β Β let pinchDist = dist(index.x, index.y, thumb.x, thumb.y);
Β Β Β Β Β Β Β Β let pinchDist2 = dist(middle.x, middle.y, thumb.x, thumb.y);
Β Β Β Β Β Β Β Β let pinchDist3 = dist(ring.x, ring.y, thumb.x, thumb.y);
Β Β Β Β Β Β Β Β let pinchDist4 = dist(pinky.x, pinky.y, thumb.x, thumb.y);
Β Β Β Β Β Β Β Β if (pinchDist < 50) {
Β Β Β Β Β Β Β Β Β Β drawingPoints.push({
Β Β Β Β Β Β Β Β Β Β Β Β x: centerX,
Β Β Β Β Β Β Β Β Β Β Β Β y: centerY,
Β Β Β Β Β Β Β Β Β Β Β Β c: currentColor
Β Β Β Β Β Β Β Β Β Β });
Β Β Β Β Β Β Β Β }
Β Β Β Β Β Β Β Β if (pinchDist2 < 40 && pinchDist4 < 40 && pinchDist > 40 && pinchDist3 > 40) {
Β Β Β Β Β Β Β Β Β Β drawingPoints = [];
Β Β Β Β Β Β Β Β }
Β Β Β Β Β Β Β Β if (pinchDist3 < 50) {
Β Β Β Β Β Β Β Β Β Β currentColor = getRandomColor();
Β Β Β Β Β Β Β Β }
Β Β Β Β Β Β }
Β Β Β Β Β Β noStroke();
Β Β Β Β Β Β for (let circles of drawingPoints) {
Β Β Β Β Β Β Β Β fill(circles.c);
Β Β Β Β Β Β Β Β circle(circles.x - vidWidth / 2, circles.y - vidHeight / 2, random(15, 20));
Β Β Β Β Β Β }
Β Β Β Β }
Β Β }
Β Β pop();
}
function gotHands(results) {
Β Β hands = results;
}
function getRandomColor() {
Β Β return color(random(255), 200, 255);
}
ml5.js sections
Declaring the handpose model and choosing the amount of hands to detect later.
Preloading the model, basically warming it up for use.
Takes the input from the video and detects the hands, which then calls a function to store it in the "results" variable.
Here we declare a constant, with each array containing a pair is where we'll draw a line connection between the two points. The numbers represent the key points of hand which the website generously provided.Β
The next block of code is in charge of connecting the key points in order to form a wireframe style of the palms. It first runs a for loop to the amount of hands, in my case: 2, then inside that for loop theres another for loop that loops until all connecting lines are generated.Β
intro & transitions
Declaring the section of the project. True = active, false = inactive. Later used if(boolean) to run sections methodically.
The last variable "NowStartDrawing" was declared because I had way too many instances where my code crashed before I could start the program since the drawing part overloaded the website.
The next block of code shows the use of millis (real time counting). I used them to make cool gradients for the title to fade in and fade out. I did this by mapping the alpha to the millis from 0 to 255 for fade in and 255 to 0 for fade out.
After that parts done, it makes the intro section inactive and then turns on the video part of the project. (The code below the if statements are just generating the text with alpha level 'alpha'.
This is the video section of the project. Basically just the code for the video to run and to where I put the hint on the interaction on the top right of the "notebook".
The video also has the fade in effect, with its own "videoalpha". However, with this one being comparatively shorter than the intro's.
visuals and decoration
Drawing the borders of the "notebook".
Borrowed code from Professor Moon that creates a row of lines that distort with the camera's video's input.Β
I modified it to be less power hungry and also made the frame rate of the camera's video show every 3 frames to reduce lag (not shown in picture).
drawing function
After the video part finishes transitioning, it makes the drawing function active. This block of code follows similar logic to the hand-key-points connection part, but instead, it takes the keypoints of 5 specific tips of each finger (e.g. pinky, index, etc.) and then calculates the distance of the 4 with the thumb.
If one of the 4 distances is less than a specific number, then it does an action.
Pinching your ring finger with your thumb changes the colors of the circles.
After the OK sign is performed (index and thumb close together), the code generates a circle with the current x and y coordinates with the current color.Β
This is then saved into an array which will later be regenerated after every frame by this code.Β
The code loops with the amount of "circles" sent to the array, each with they're exact position and color assigned.
Reflections and Improvements
This project felt like a completely different one from the original proposed version.
In the proposal, I wanted to create something that could be saved, even if the program was closed or opened in another computer. The basis of my proposal was being able to leave your trace, leaving behind your emotions, thoughts, 'you' in an interactive canvas. However, as I coded and changed paths with comments from the professor and peers, It turned into something more similar to a ml5.js showcase. In the IMA Show, Professor Moon's words resonated with me, I didn't see much 'me' in the project. It was very minimalistic and most code had few to no addition from myself. Still, the feedback I received from the fellows and many friends along the way made me realize that the mixture of all these codes, although some of it wasn't coded by yourself, is still an artwork made by you. With this thought in mind, I felt relieved that my project's functions, interactions, concepts, basis wasn't so deviated from its title: You Are Art.
However, this still doesn't mean that the code is perfect. I still see many improvements that would work well if added.
One thing that stood out to me during the feedback and user-testing day was when Professor Leon said that it'd be nice to see more movement from the circles generated. "Maybe make it so that when it reached a hundred circles then they'll all bundled up at the center or something". Yes, I did want to add more to my project, and I feel due to the openness of my proposal and the amount of playing around I did with the ml5.js models, I lacked the time to be able to add to the project. Aesthetic-wise, there were few things I could've fixed, many have told me that the instruction's sentence was a little bit small and hard to see so most people missed it and didn't know what to do with the notebook. I think fixing its position and text-visibility is a good start for improvements.Β
Nevertheless, this project was very fun to explore and I thank Professor Moon for introducing these amazing concepts in Creative Coding Lab!
CreditsΒ
Although I didn't book a lot of office hours with Professor Moon, he still helped me a lot with the direction of the project. The background's distorted lines was also coded by him, credits to the Professor!
I used to AI to give me suggestions on how to handle errors or obstacles. An example of this is the issue of the circles disappearing since the background's color refreshes after every rate. ChatGPT suggested I store the circle's position and color in an array and then regenerating them every frame.
https://docs.ml5js.org/#/reference/handposeΒ
Finally, we have the ml5.js machine learning model: handPose, developed by people at NYU and also Professor Moon! The model was borrowed and used for the interaction of my project. The documentation posted above this also helped me learn the setup and how to play around with the model.
This project draws a lot of inspiration from "We Feel Fine" by Jonathan Harris