Final Goal
The deliverable for this project is being able to ask Shimi to "play"/"perform"/"improv"/etc. with you, Shimi listening to your music and, in real-time, playing with you.
Week of February 1, 2019
- Connected to the gatech VPN
- Installed PuTTy
- Installed Xming
- Brainstormed incremental steps to final goal
- Shimi needs to listen for one pitch (frequency)
- Then listen for beat / rhythm
- Then listen for multiple pitches at the same time
- Then listen for chords
- Then listen for key
- Shimi needs to analyze the inputs and respond accordingly in its own voice
- First step -- create functionality that allows Shimi to respond (make a sound) upon hearing a sound
- Then respond in a different synth sound upon command
- Then respond with an added sample
Minutes from first meeting of the semester with Shimi team
- Notebooks
- conda: virtual environment for python
- allows to have different versions / different package requirements for python
- can experiment with pyo
- using PuTTy
- for interfacing with Shimi
- no accessible video output
- need to use a SSH client -- a remote login program
- refresh on linux command line
Weeks of February 8 & 15
- Added to Shimi project Github repo
- Set up remote host in PyCharm to work with Shimi
- Refreshed on Python with simple programs
- Installed pyo module for signal processing
- Familiarized with relevant libraries in pyo module
Weeks of February 22 & March 1
- Implemented Shimi’s listen and response functionality
- In pyo, use Input object for handling real-time input from a microphone
- Use Yin object to analyze that input (pitch tracking)
- The audio output of the object is the estimated frequency, in Hz, of the input sound.
- Use Sine object for Shimi to play a sound of the frequency detected with Yin
- Tested locally and with Shimi's Jetson processor
Weeks of March 8 & 15
- Exploring next steps…
- Polyphonic pitch detection (handling multiple pitches at once)
- Smoothing / filtering of pitch to reduce white noise
- Research moving average filter
- Pattern object in pyo allows getting a window of pitches over time, can use this for a moving average
- Use logarithmic smoothing so Shimi weighs more recently heard sounds more heavily
- Start with simple moving average
- Make more unique sounds for output besides simple sine wave
- Beat / rhythm detection
Week of March 22
- Prepare presentation for ¾ VIP presentation
- Test demo
Week of March 29
Weeks of April 5 & 12
- Implement moving-average filter to smooth playback
- Fine tune on/off detection so playback strictly occurs only when user sings
- Prepare for final demo presentation
Week of April 19