Music on the Body
Music Visualizer
Embedded Files
A different way to expereince music
Final Video Header.movIntention and Inspirations
Intention and Inspirations
This project came from a curiosity I've had for a while now. I have often thought about how can we represent music through light and possibly person recognition. However even typing that seems like an impossible task. I had thought of different versions of this particular project a few time over the years and thought maybe I can try to attempt one of the versions now. A Music Visualizer alone would not have been challenging enough for me, so I found it to be a great opportunity to bring this long awaited curiosity to life once I saw what Processing could do. I had a lot of inspirations for this. They range from films, live concerts I've attended, as well as a company called Active Theory that I keep up to date with. Even my Phillips Hue Sync Box for my TV is a large inspiration because of the way the rope light enhances my viewing experience, I thought maybe the same can be done with this.
This project is something I can build on, and I coded some things purposefully so I can continue working on this. I can see this being used a parties or events as an interactive experience, even with multiple people. I believe this has a lot of potential to grow into something much cleaner and customizable too!
I listen to a lot of music. My most recent Spotify wrapped said I had over 100K minutes of listening time. Music helps me focus, clean, sleep, and bond and I feel that this project can add an extra layer to all the emotions that are triggered while listening to music for different people.
Concept
Concept
To Summarize How This Works
A live camera feed will be read to separate the background from the foreground to create a smooth silhouette.
Computer Vision is then used to detect motion and create contours around the moving object.
Get points of the contour and map a shape (in this case circle) to represent the contours.
Use FFT to get information about the input coming in through the mic
Map a frequency band to the size of the circles so they shrink and grow with the sound.
How the User Interacts and Best Practices
As the program starts, you will need move out of view and click to create a artificial 'green screen' for the motion detection to work better. Like to the right.
BEST PRACTICES and TIPS
Be sure to have even lighting
Use a blank wall - this helps make sure the camera is only detected changes by you, and not your window for example
The sound is based on microphone input, so sounds other than music may effect the way the circle are represented.
If youre using a mac you can enable background as a shortcut if you do not have a blank wall. (I show both below)
2. Once the background is created, all theres left to do is play music and the circles will generate on their own.
**Enable Background on Mac, if you do not have a blank wall**
Step 1 - create background.movBreaking Down the Code
Breaking Down the Code
I made an effort to explain a lot of my code within the comments but there is a lot more that needs explaining.
Root / Main
Root / Main
In the root file, I used this as the base skeleton for the project, to keep things as tidy as possible. Here is where I needed to create all my variables, objects and get all my library imports. Inside draw() I had to read each frame from opencv and, remove the background and apply the circles after getting the points from the contours.
In order to create the 'green screen' effect there is a mousePressed function that triggers a copy function and when the screen is clicked so we have an image to compare the motion detection to.
This file uses:
Variables
Functions
Loops
CHALLENGES
CHALLENGES
Throughout this project I have saved about 10 different versions because I had no clue how to use the OpenCv library. My most difficult problem was getting the opencv stream to be the only one that displayed. Referenced in our Functions (Week 9) you can see 3 different video feeds coming through, and remove background was not working on the OpenCV object and only the Capture stream through Processing. This was all because of types!
OpenCV is of OpenCV type and Processing needed a PImage to manipulate. I knew I needed to convert opencv to PImage, so I tried using Mat Objects, a method called toPImage() and toCV() in the OpenCV for Processing Docs. None of it worked, this went on for weeks and I considered changing my topic. After some time away from the project I gave fresh eyes to the documentation and saw getSnapshot() was of type PImage. My logic was if we are processing videos one frame at a time, I thought maybe I could use them interchangeably and it would inherently change the type from OpenCv to PImage if I did.
opencv.getSnapshot()
This solved A LOT of issues for me! I was able to manipulate the frame of the opencv object using methods and functions meant for PImages. These challenges also contined in the Remove Background Function.
Like I have mentioned a few times I referenced a lot of Daniels code. This is the remove background function. In my mind, this function helps create a smoother silhouette of the object. Computer vision can still sense movement without it , but it would not have as confident an outline.
The function loops through each pixel of every frame, and gets the 1D (one dimensional) location of the pixel. Then we can get the color of each pixel at their location. Th reason why we do this is so we have comparable items to decipher what is the background and what is the foreground!
in order to do this we need all the pixel colors for the live feed and the background image we take using mousePressed in the root file,
Once we have the colors we can compare the pixels to see if it is different enough to be considered the foreground. This is why is it important to have a blank background and good lighting. An unpredictable background can cause there to be detected 'movement' in the background if the pixel is registered to be different enough, or like in the code if the difference is greater than the threshold.
This file uses:
Loops
Variables
Arrays
Functions
Conditinal Statements
CHALLENGES
CHALLENGES
Because this function was not made with Computer Vision in mind, it was extremely difficult for me to understand how to incorporate it so the background removal was happening to the OpenCv object and not the Capture stream object. The pixels[] array is for PImages, and I had the same issue as I had above with trying to convert the opencv into PImage. but getSnapshot() help with this problem as well and I was able to cycle through the pixels and have only one video feed instead of three. Knowing where to use frame was intresting because it does not like frame.updatePixels, but it did like frame.loadPixels. There was a lot of times in the project where the order of how things are places really can make or break the functionality, just like loadPixels.
Remove Background Function
Remove Background Function
Show Contours Function
Show Contours Function
This is perhaps the most fun function. This is where the contours are found and the circles are added around the outline of the person.
We begin by using the code to findContours from OpenCV for Processing. While looking at the documentation I saw a getPoints method and felt it was appropriate to use to get the information I needed to manipulate those coordinates. Then the points are saved in an array so I can pull the data out for the X and Y's.
Circles are them drawn at the same locations.
As for the size of the circles that is purely based on the sound input. In the root file, we set up what we needed to analyze the sound input from the mic. With that sound input it is mapped to an array of 16 bands and each band will represent a different frequency of sound coming through. I specifically chose 16 bands because the docs mention it needs to be a square root of 2 and 8 seemed like too few. I looked at each band using spectrum[band number] and found one I liked and seemed to captures the essence of the beat the most, which was spectrum[2]. The number then needed to be scaled up because it return small numbers.
This file uses:
Loops
Arrays
Variables
CHALLENGES
CHALLENGES
I had a theme of type conversions being a major issue for me. I was not very familiar with PVectors and the getPoints returned a PVector ArrayList. Seeing how little code all this turned out to be makes me frustrated that it took so much trial and error to get something back and realizing it was maybe just one line of code. Like the other files I had data I needed to get to but didn't know how to reach it. For this it was the X and Y coordinates of the contour points. I tried using .get().get() but it did not work the same in Processing as it would have in Java, as well as .toArray().
Once I noticed the conversions in Processing, it made it much easier to work with.
Working Demo
Working Demo
My brother helped with being a stand in for the project
I am also demonstating the difference in the silhouette when I use the function
versus when a virtual green screen is being used,.
Final Working Demo.mp4Page updated
Google Sites
Report abuse