3. Sound Bubbles

Motivation

My inspiration was synesthesia, specifically chromesthesia. It refers to the involuntary association of heard sounds to colours, shapes and movements.  Since music and sounds, in general, are temporary and are quick to fade from space, I wanted to make something that allowed the sounds or the memory of them to remain in the space, presented linearly so that an entire sound sample could be experienced in one go. I wanted a visual representation of each moment in a sound sample, but I also wanted people to be able to actively interact with it, and see how the sounds they make look, or how the sounds in their environment look. I wanted to take into account power, pitch, and other features of the sound, and map them to visual features like sharpness, transparency, size and colour.

Implementation

For this project, I used openFrameworks. I used 2 downloaded addons- ofxAudioAnalyzer and ofxAudioDecoder. AudioAnanlyzer breaks down sounds by predetermined duration lengths and calculates a variety of things about the sound, like energy, pitch frequency, dissonance, strong peaks and decays, etc. It uses the audio decoder add-on to open sound files and make these analyses. I used one of the examples in the addons to see the range of values that each analysis provides and to understand how to set up the code to allow analysis. I started working around an mp3 of a song, figuring out factors based on how I saw moments in the song. I later added a key to allow for live analysis and a few different sound samples for demonstration purposes.

Check the drive for code and used files.

https://drive.google.com/open?id=1ZR2sOl3JKWvygJiVnnYg_egyBnHwfqMn

Future Ideas

While I was able to get the sharpness, transparency and size to work how I wanted it to, I know I could do more with colour.  With more time with the platform and my chosen addons, I would have liked to play around with what the colours represent, perhaps by mapping darker colours to minor chords, or allowing colours to be stand-ins for emotional valence- yellows for happier sounds, reds for more passionate sounds, and greens and blues for more calm moments.

Another idea that may be interesting would be to play with the x and y position of the drawn objects. Right now they are being plotted linearly, but when artists with synesthesia make paintings around musical pieces they generally convey the piece as a whole, adding shapes as they appear in space when they hear the sounds. It has the added effect of conveying movement. It would be interesting to play around with making song based artwork non-linearly.