Project 5

WaveNet is a deep generative model of raw audio waveforms. It has exceptional performance in voice generation and music making. My goal was to use WaveNet to generate raw audio. I recorded myself playing Estampes by Debussy for 45 minutes, and I want to generate raw audio using that recording. I trained the network for 48 hours for 30000 iterations. The result, at the first glance, was quite disappointing. Yet I found the audio interesting in its repetition of certain pitches that are representative of the piece I played. Therefore, I decided to collaborate with the network in generating an ambient soundscape.

Instead of using some existing text-to-speech dataset, I recorded my piano playing using a Shure portable microphone. I did noise reduction on the recording using Audacity. I trained a WaveNet network on this 40-minute recording. On the output audio, I did noise reduction by 33 dB, time stretching by 10 times and editing using Audacity. Also, I did granular synthesis on the processed audio using Ableton.

Infinite Descent: A Theoretically Great Idea was inspired by my experience as a TA for the course 15-251: “Great Theoretical Ideas in Computer Science.” I trained an LSTM on the course notes, recitations, and homework assignments, and then I had it generate long passages of text. Half of these I transcribed by hand to resemble the proofs that students submit for homework and TAs must grade. The other half I formatted to resemble the course notes, which students study to learn the material. In both situations, the reader is motivated to understand what is written. This piece invites viewers to step into these roles and experience the frustration of never fully comprehending the text.

Landscape design addresses the composition of gardens and public green spaces with elements such as plants, furniture, small constructions, etc. The composition of plants is a particular challenge for design, as their clustering and grouping in masses involves associating different features of their complex appearance and behavior.

Incrementally the system adds n new tree genes to the set X. Each tree gene is defined by 7 parameters: scale between branches, random, max height, angle of branches, n of branches in division, n of generations of branches, probability of extending the trunk. The addition of new genes is followed by the embedding of the genes of all the current set of trees (X) in a bidimensional space using specific parameters (n_components=2, perplexity = len(self.X)//20,

early_exaggeration = 100 + len(self.X), n_iter = 500). The embedding decides where the trees are going to be located in the current garden, When the embedding process finishes, this new garden is generated in Rhino via UDP connection with GH Python/Grasshopper. The user can visualize multiple gardens been generated by the clustering of their formal characteristics.

This project explores the possibilities of integrating the RNN-Sketch Demo with a robotic arm to form a collaborative workflow between user, RNN, and robot. The motivation behind this project is to find a robot-human interaction scenario to open the discussion for further development in Robot-Art competition.

The model consists of two main component, one is the grasshopper definition and embedded Python scripts that handles image processing, toolpath generation, simulation, and communication with the robot. HAL add-on for Grasshopper is the core engine behind the robotic control and simulation. The other component is Sketch_RNN code that generates the doodle based on the user drawn seed. It is using the Magenta library.