WIIMOTE Controller

Motivation/Background

My inspiration from this project came from two places. The first is artist Imogen Heap's Tiny Desk Concert. In the video, she demonstrates her Mi.mu gloves, which are (according to the website) "the world’s most advanced wearable musical instrument, for expressive creation, composition and performance." For this performance, Heap shows how the gloves can transform her voice with looping, pitch shifting, reverb and other controls that she can manipulate using just these gloves. Another performance with these gloves that served as an inspiration was Ariana Grande's use of them. Grande performs the same song as Heap ("Hide and Seek"), but uses the gloves to harmonize her voice, alternating between four positions and four chords. Both these performances served as the inspirations for the functions of my controller. As for the inspiration behind the actual device, I took MUMT 302 last year and we also had final projects. One of the examples for the projects was someone who made a controller out of a XBox Kinect Console. I didn't have an XBox, but I did have a couple of Wii remotes (Wiimotes) laying around and I had done some initial research and saw that it was possible to connect them to your computers. So, I decided to create a music controller similar to the gloves ( though much simpler) using the Wiimote.

Objectives

My goals was to create a Max patch that could be (almost) completely controlled using the Wiimote. I wanted to implement functions, similar to the Mi.mu gloves that could control a live input. My ultimate goal in mind was for a vocal performance. It couldn't be completely controlled by the remote because there would be some programmable attributes, mainly the chords used for the harmonizer. Once the user chose these, evrything else can be controlled with the Wiimote. The position of the Wiimote would control the chords for the harmonizer. Then, there would be a simple four track looper, reverb control, delay, and chorus. All of these could be changed live.

Interpreting messages from Darwiin Remote in MAX

Darwiin Remote OSC

Interpreting the messages in Max

The WIIMOTE and Darwiin Remote OSC

The Wiimote is used as an accelerometer and also has 11 buttons that can be used to control it: up, down, left, right, A, B, plus, minus, home, one, and two. (There is also a power button but I was not able to access it). It can communicate in two ways, IR and via Bluetooth. I wanted to find a way for it to communicate with Max. I did some research and found the software Darwiin Remote. According to it's SourceForge description, "DarwiinRemote is a tiny program & library which reads data from and sends data to Nintendo Wii Remote (Wiimote), nunchuk, & classic controller." It connects the Wiimote over a Bluetooth connection. Then, matthiaskronlachner updated it so that it could communicate via OSC. This was the version I used for my project.

Then in Max, to interpret these messages, I used udpreceive and made sure that the port was the same as in DarwiinRemoteOSC. There was a provided file in the download for the program called "AddressSpace.txt". This gave me the messages that would be sent to Max. The ones I cared about came in two forms: wii/button/a b, where a is the name of a button and b is 1 or 0, and wii/orientation x y z, where x is the x rotation, y is the y rotation, and z is the z rotation. I unjoined the messages, and then sent the messages (s wiimessage) and used the match object to find corresponding messages. I looked for seperate buttons and orientation.

I then made a p Trigger patch, because each time I pressed a button , it banged twice(press and release). The subpatch sent out one bang for every two bangs received. Then, these bangs could be sent to other parts of the patch to control other parameters.

Harmonizer

I wanted to keep the harmonizer simple and limit it to four positions. That way, the ranges for the positions could be well defined and it would be easier to control. It took a lot of messing around, but I eventually found four different positions that I could use. They are defined in the following way:

Rotate Right
- -45 < y <45, x < -45
Rotate Left
- -45 < y < 45, x > 45
Up
- -45<x<45, y > 45
Down
- -45<x<45, y <-45

This gave me up to four chords to use. I created a patch called "chords" to control the harmonizer. As input, it takes the four chords chosen by the user, the wiimote position as well as a major key. The chords are hardcoded into coll Shift, which defines the intervals of the chord in relation to the tonic (i.e. a V chord is defined as 7, 11, 2). Then, depending on the position, the intervals are added to the tonic, scaled by 36 (to get into a range of 1-12), and then sent to three seperate retune~ objects (root, third, and fifth of the chords). Given a note, the retune~ object changes the frequency of the incoming signal into the closest octave of that note. The entire harmonizer can be triggered on or off using the home button.

The "chords" subpatch

Interpreting the four different positions of the wiimote

The Looper Patch

Looper

To build the basic looper, I got the help from a youtube series linked here. The youtube video built a looper that would have a user select a channel (1-4). Then it would record to that channel and loop that audio clip. For each channel, it opens, makes, and/or replaces a wav file (LoopX.wav) using sfrecord~. It also stores the total time for each file. Then it's looped using the groove~ object for that total time.

I wanted my looper to be able to be controlled using only two buttons (A and B). So, I had to implement some triggers that were dependent on the end of the loop. To use th looper, a user presses A to trigger the first loop to start recording. Then, when A is pressed again, the loop stops. The time of this loop becomes the time for all of the loops. The first loop continues to play. Then, when A is pressed again, the 2nd loop starts recording only for the time of the first loop. Then it starts looping and is the same length as the first loop. Then this repeats for loops 3 and 4. Pressing B stops all of the loops.

Delay, Reverb, Chorus, Gain

As for my last three controls, I was left with the left,right, up, down, 1, 2, +, and - buttons. I wanted to implement simple controls that could easily enhance the sound, without adding too much complexity. So I settled on four controls: Gain, Reverb, Chorus, and Delay.

For gain, I wanted a pretty intuitive control, so I used the + and - buttons. The delta for each increment or decrement is 10 to allow each change to be signficant.

Reverb is controlled by the up and down arrows. I used the yafr (yet another free reverb) object and again incremented/decrement by a factor of 10 to change the reverb time.

I found a Chorus object online (which is controlled by 1 and 2). Once again, it is scaled by a factor of 10 and controls the amount of wet vs. dry signal being sent ot the output (incrementing increases the amount of wet signal).

Finally, I implemented delay using tapin~ and tapout~, controlled by the left and right arrows. This time, it's incremented/decremented by a factor of 100.

Gain Control

Reverb Control

Delay Control

Chorus Control

The Interface

The Four Chord Positions

The Looper in action

Interface

The last thing left to do was create an interface that was easy to understand and simple. I came up with a design that includes programmable attributes (for the harmonizer), control displays, and a wiimote. The wiimote includes feedback (red button flashes) to show a user that their messages was received and the program is working.

As for the programmable attributes, before performing, a user would have to select the key of the piece, as well as choose up to four chords that they could include in the harmonizer. The user also manually has to turn on the ADC so the patch can receive input. Other than that, everything else can be controlled entirely from the wiimote.

When a button is pressed, the corresponding control value is changed. The current chord displays the chord that is being played (based on the position). Finally, the user can see their looper represented visually with four circles, the outer one representing the first loop.

There are also some instructions on how to use the patch, including what all four chord positions look like, and how to us the looper.

Video Demonstrations

Demonstration of the OSC Messages:

OSCDemonstration.mov

Demonstration of the Patch:

PatchDemonstration.mov

Demonstration of the controls (Gain, Reverb, Delay, Chorus, Looper, Harmonizer)

Finalproject mumt306.mp4

Demonstration of the harmonizer (performing Only One by Kanye West and Paul McCartney):

OnlyOne_Harmonizer.mov

Limitations

Throught this project, I found that there were a couple limitations due to the nature of the wiimote.

First of all, the harmonizer I have created is limited due to the fact that there are only four positions. There is the possibility to add more positions, but that comes with the risk that the actual threshold of the position becomes very small and it becomes harder to control. There are also a limited amount of chords to pick from, but that could easily be expanded. More chords would just have to be hardcoded into coll Shift.

Another limitation was the amount of controls I could have. Because I wanted to be able to increment and decrement my controls, I needed two buttons for each one. Since I only had 11 buttons to start, I could have up to five controls and hand one left to turn on and off the harmonizer.

Also, the looper is pretty limited in its implementations. There are only up to four loops allowed and there is no way to turn back on a loop once it's been turned off. Either the loop has to continue, or it gets erased and overwritten.

I also didn't use the acceleration of the wiimote as a parameter, though that could be added to more dynamically change controls.

Conclusion

Though this patch was limited, I can see it easily being expanded in the future. Essentially, all that needs to be done is create patches that can be controlled by a button. Then, that button can be assigned to any corresponding button/buttons on the wiimote. Also, adding more positions could allow for even more controls (compared to the 15 that I had implemented here). Throughout this project, I was able to familiarize myself with UDP and OSC messages and controls. I also had the opporutinity to learn about some new max objects, notably retune~. It was a challenge to get a lot things to work, especially the looper, but in the end, I was able to achieve my initial goals.

It is also extremely fun to play with this project and allows me to perform without using instruments or other controllers in a slightly more expressive way (but not as much as the Mi.mu gloves). All in all, I'd like to keep expanding this patch and seeing how far I can take it for a more expressive performance.

Downloads/Links

Darwiin Remote OSC: http://www.matthiaskronlachner.com/?p=1574

The Patch: https://drive.google.com/file/d/1_jfidQ6h_PPu9PREgVFpNAqPti3zzTLW/view?usp=sharing

Page updated

Google Sites

Report abuse