Final Goal
The deliverable for this project is being able to ask Shimi to "play"/"perform"/"improv"/etc. with you, Shimi listening to your music and, in real-time, playing with you.
Week of February 1, 2019
- Connected to the gatech VPN
- Installed PuTTy
- Installed Xming
- Brainstormed incremental steps to final goal- Shimi needs to listen for one pitch (frequency)
- Then listen for beat / rhythm
- Then listen for multiple pitches at the same time
- Then listen for chords
- Then listen for key
- Shimi needs to analyze the inputs and respond accordingly in its own voice
 
- First step --  create functionality that allows Shimi to respond (make a sound) upon hearing a sound- Then respond in a different synth sound upon command
- Then respond with an added sample
 
Minutes from first meeting of the semester with Shimi team
- Notebooks
- conda: virtual environment for python- allows to have different versions / different package requirements for python
- can experiment with pyo
 
- using PuTTy- for interfacing with Shimi- no accessible video output
- need to use a SSH client -- a remote login program
- refresh on linux command line
 
 
Weeks of February 8 & 15
- Added to Shimi project Github repo
- Set up remote host in PyCharm to work with Shimi
- Refreshed on Python with simple programs
- Installed pyo module for signal processing
- Familiarized with relevant libraries in pyo module
Weeks of February 22 & March 1
- Implemented Shimi’s listen and response functionality- In pyo, use Input object for handling real-time input from a microphone
- Use Yin object to analyze that input (pitch tracking)- The audio output of the object is the estimated frequency, in Hz, of the input sound.
 
- Use Sine object for Shimi to play a sound of the frequency detected with Yin
 
- Tested locally and with Shimi's Jetson processor
Weeks of March 8 & 15
- Exploring next steps…- Polyphonic pitch detection (handling multiple pitches at once)
- Smoothing / filtering of pitch to reduce white noise- Research moving average filter- Pattern object in pyo allows getting a window of pitches over time, can use this for a moving average
- Use logarithmic smoothing so Shimi weighs more recently heard sounds more heavily
- Start with simple moving average
 
 
- Make more unique sounds for output besides simple sine wave
- Beat / rhythm detection
 
Week of March 22
- Prepare presentation for ¾ VIP presentation
- Test demo
Week of March 29
Weeks of April 5 & 12
- Implement moving-average filter to smooth playback
- Fine tune on/off detection so playback strictly occurs only when user sings
- Prepare for final demo presentation
Week of April 19