• MEAPsoft FFT: Java implementation of the Fast Fourier Transform algorithm

• AudioRecord: signal sampling library

• HBRecorder: screen recording library

• Also see ssaurel's tutorial for a touchscreen piano application, from which I adapted my PianoView for the main GUI

• This application makes use of the device microphone

The signal sampling and processing involved in this application starts with the creation of an AudioRecord object. This object requires a sampling frequency and the size of a buffer, both of which varies from device to device and are determined using built-in methods at run-time. The sampling frequency is made at least twice as large as the maximum frequency that we want to find (in this case ~ 1 kHz). Another array, we call x[n], reads the samples from the buffer, where the length of x[n] is the closest power of 2 that is larger than the AudioRecord buffer size. The power of 2 requirement comes from the decimation procedure of the FFT.

From the length of x[n] (N) and the sampling frequency (fs), we can determine the frequency bins of the resulting FFT. The frequencies will range from (-fs/(2N)) and (fs/(2N)). The frequencies of the keys on the PianoView, which start at middle C (~261 Hz), can be derived by the formula 2^((m -49)/12) * 440 Hz, where m is the key number of an 88-key piano. Before we take the FFT, we determine the mapping from the FFT bin frequency and the actual key frequency by finding the FFT bin number k that has a bin frequency closest to each key's frequency in the PianoView (m = 40 to m = 63).

The next step is to perform FFT, which is done with an FFT object. The fft method is called on x[n] and an array of zeros of the same length y[n]. x[n] and y[n] represent the real and imaginary parts of our signal. After the method call, these two arrays which are retained in memory now hold the real and imaginary parts of the spectrum of the sampled signal, and we will now call these signal X[k] and Y[k]. Now we use the mapping done previously to use these components of the spectrum to update the spectrum_magnitude fields of every key object that is contained in the keysList ArrayList (which represent each key on the PianoView).

Next, the keys in the keysList have their spectrum magnitudes normalized to the range [0.0, 1.0], while also calculations are made on the entire set to determine distance between the max and the min, mean value, and standard deviation. If the distance between the max and the min is smaller than some empirically determined threshold, then the signal is considered as noise/silence, and so the next steps are skipped. Otherwise, we take one of two paths. If the singleMode is true, then we process the signals by calculating the correlation coefficient of the spectrums calculated and spectrums in a look-up table I previously recorded and that correspond to each possible note on the PianoView. The key with the highest correlation coefficient is added to keysToUpdateFirstSet ArrayList. If singleMode is false, the keys with a spectrum magnitude greater than half a standard deviation above the mean are added to keysToUpdateFirstSet.

The sampling and processing steps occurs once more, and the results of keysToUpdateFirstSet are copied to keysToUpdateSecondSet while the first set receives the new values. If the first set and the second set are equal, then we update the keys on the PianoView. This two set process is done to reduce unwanted peaks from noise and harmonics to be reflected on the PianoView. To update the keys on the PianoView, the PianoView instance has an updateKeys() method that takes in an ArrayList of Key objects. Inside PianoView's updateKeys(), all keys in another arraylist called keyGraphics have their pressed field set to false. Then, the keys received from the update list have their pressed field set to true. In the end, the canvas is invalidated and so the method onDraw() is called, which redraws all the KeyGraphics on the canvas, with the ones that have pressed as true colored blue to indicate that they were pressed.