Liana

Week of February 1, 2019

• Created incremental steps for Shimi gestures
  • I want Shimi to be able to detect the different part of the music (vocals from the instruments). Different parts of Shimi should move corresponding to these different parts.
  • I want Shimi to be able to tap to the down beat of the music, switch to a syncopated beat, and then go back to the down beat.
• Got access to the GitHub and looked over shimi.py, motion/move.py, and audio/audio_demos.py
• Connected to the VPN.
• Installed PuTTY.

Week of February 8, 2019

• Installed Xming.
• Learned how to SSH into Shimi's computer.
• Set up PyCharm remote host
• Created audio/tapping.py in my directory

Week of February 15, 2019

• Created play_gladys in audio/audio_demos.py
• Sampling rate

Goals for next week:

• Look into Librosa for tempo and beat detection
  • onset
  • load
  • pass samples and sampling rate into tempo
    • seconds per beat
    • 1/value then get inverse; 1/(1 beat/60 second)
• Source separation
  • Wave-U-Net
• Look into Shimi responding/gesturing to a song in real time
  • Spotify's audio analysis
  • Close to real time. Do audio analysis on .5 seconds and determine next move.

Week of February 22, 2019

• Wave-U-Net
  • For vocal separation, need to download CCMixter as well.
  • pretrained models already or can train models using the given datasets
• Installed Librosa on my local device
• Spotify Audio Analysis
  • used the demo found on the website
• Goals for next week
  • pitch chroma: spotify and librosa

Week of March 1, 2019

• edited play_gladys in audio/audio_demos.py to use librosa.beat.beat_track
• goals
  • demos: play through slides
  • tempo detection solidified
  • Wav-U-Net

Weeks of March 8, 2019 and March 15, 2019

• used the pretrained models in Wave-U-Net to create song separations locally
  • vocal and accompaniment
  • multiple instrument
• goals for next week
  • demo for 3/4 presentations
  • have shimi move to the voice separation
  • set up meeting with Ryan to get Wave-U-Net working on the Jetson or SD card

Week of March 29, 2019

• worked on 3/4 presentation

Week of April 5, 2019

• researched other manners of audio separation
  • https://arxiv.org/pdf/1803.00702.pdf
    • "introduced a novel multi-resolution convolutional auto-encoder neural network to separate the stereo waveforms of the target sources from the input stereo mixed signals"
  • https://www.tandfonline.com/doi/pdf/10.1080/01969720290040777
    • monophonic vs. polyphonic needs to be treated differently. The latter uses "blackboard model or multiple-cause sparse coding method."

Week of April 12, 2019

• Away for conference for Management Leadership for Tomorrow.

Week of April 19, 2019

• prepared final presentation demo