Nintendo's ultra–affordable Wii Remote controller can sense movement in every direction, and even knows at what angle you're holding it. Its potential as a music controller is unlimited, and you don't even need a Wii console to use it!
What are the musical instruments of the future? Are we going to be using keyboards, guitars, saxophones, and drum kits forever? Thanks to digital technology, what an instrument sounds like doesn't have to have any connection with what it looks like or how you play it. So should we now be making rather more interesting choices about what constitutes a 'musical instrument'?
There's a vast international culture devoted to thinking about these questions and devising possible answers to them, and they meet every year at the New Interfaces for Musical Expression (NIME) conference, which I reported on in the January 2006 issue of Sound On Sound (www.soundonsound.com/sos/jan06/articles/nime.htm). Many in that group make their own interfaces, while others like to use off–the–shelf hardware borrowed from other areas, especially toys. These days much of the buzz, in the NIME crowd and among hackers all over the world, is about Nintendo's wireless Wii Remote.
Introducing The Wiimote
The Wiimote, as its friends call it, is arguably the greatest achievement in game controller history. First announced in September 2005, the Wiimote didn't ship until 14 months later, and for at least a year following the first shipments, the demand far outstripped the supply. And small wonder: for a mere £30 or so, the Wiimote provides interface power that just a couple of years before would have cost literally thousands. In a package the size of a hot dog, it combines 12 buttons, two infra–red sensors for two–dimensional position sensing, and an accelerometer that provides three–dimensional motion detection. It communicates with its base station using standard Bluetooth technology.
If you've played some of the more physically oriented games now available for the Wii, you are probably aware of all this, and perhaps you may have thought about what it could mean if all this power were to be channelled into making music. Two–dimensional position tracking allows you to point to an object, real or imaginary, and the Wiimote will know precisely where you're pointing. The accelerometer is an Analog Devices ADXL 330 — which, if you were to buy a single unit, would cost 50 percent more than the Wiimote itself. (Nintendo reportedly pay about $1.50 each — but they buy millions of them.) It has a sensitivity of 3G, or three times the force of gravity, which means that while it won't do too well if you slam it into a wall, it is sensitive enough to measure the force of gravity itself. Thus it not only detects motion, but it also reports the angle you are holding the Wiimote at when it is not moving — and not just one angle, but three: vertical, horizontal, and rotational. In the nomenclature of flight dynamics, it detects pitch, yaw, and roll.
But that's not all. An accessory to the Wiimote is Nintendo's Nunchuk, which adds two more buttons and a joystick, as well as another three–axis accelerometer, for a mere £15 more. Another accessory, which is even less expensive, is what Nintendo call a 'Sensor Bar', which enables the infra–red tracking function I mentioned earlier. (The name is a complete misnomer: the bar doesn't sense anything, it emits infra–red light that the Wiimote senses.)
Among musicians, all of this would be of interest only to those who already have a Wii system, except for one significant fact: you don't need a Wii system to use the Wiimote. The Wiimote talks Bluetooth, that ubiquitous 2.4GHz wireless standard, and therefore can interface all by itself with any computer that also supports Bluetooth, either with the chip set built into it or using one of the many inexpensive USB dongles available. As for the Sensor Bar, a number of third–party manufacturers have come out with 'wireless sensor bars' (which are nothing more than infra–red emitters powered by batteries) that don't need a Wii system either.
Making Music
So: once you get information from a Wiimote into your computer, what can you do with it? A whole lot, it turns out, and if you search on YouTube you'll find that there are a huge number of people working on making music using Wiimotes, many of them in pretty silly and mundane ways, but many others very cleverly.
But just because the signal formats match, this doesn't mean that a Wiimote can automatically play your computer synth. First you have to figure out how to convert the Wiimote's various data streams into a form that can control music–generating software or hardware. One approach is to use a general–purpose interpreter that translates Wiimote parameters into standard computer gestures, such as mouse movements, clicks, drags, menu actions and key presses. Remote Buddy, a Macintosh utility created by the German company IOSpirit, is an example of this. Priced at 20 Euros, it lets you create 'behaviours' and scripts that can do almost anything in almost any Macintosh program (and it works with a wide variety of other hardware aside from Wiimotes, including EyeTV remotes and Bluetooth–equipped cellphones). So for example, a musician could use it to control tracks or loops in a virtual mixer, or play notes on a virtual keyboard, or select and trigger drum notes and patterns.
On the Windows platform, a popular general–purpose tool is GlovePIE, a free scripting language by Australian programmer Carl Kenner. Originally developed for use with virtual–reality gloves (hence the name), GlovePIE now supports all manner of game pads, trackers and joysticks, and also Wiimotes. It comes with a library of scripts, and you can easily create your own, using either command–line or graphic interfaces. GlovePIE is especially interesting to musicians because Kenner includes a set of MIDI commands that work through Windows' MIDI Mapper, and also commands for Open Sound Control (OSC), the communications protocol developed at the University of California Berkeley for sound and media processing, web interactivity, sensor input and other high–level applications.
Other useful utilities are the free Wiinstrument, for Mac and Linux, by Tobias Moebert and Sebatian Burkhart, and Wiinremote for Windows, from Japanese developer Tokkyo Nakamura.
Being able to move things around on a screen while waving a Wiimote in the air is a lot of fun, but serious music applications need multiple parameter control, and the Wiimote translators that speak MIDI fill that bill nicely.
OSCulator is one of several utilities that can translate Wii Remote data into MIDI data, which can then be used to control almost any music application.My favourite way to use Wiimotes for music is with a Mac program called OSCulator, written by a Paris musician and programmer named Camille Troillard and distributed as 'donationware' (minimum contribution $19). Camille is a devoted user of Symbolic Systems' Kyma, and he wrote OSCulator to be able to control his Kyma with various external hardware controllers, such as Wacom drawing tablets, 3Dconnexion's SpaceNavigator, WiiGuitars and, of course, Wiimotes. Like GlovePIE, it provides OSC and MIDI outputs, but it has a wonderfully simple and efficient user interface. It lets you map all of the controls on as many as four different Wiimotes to MIDI commands on any channel, and its setup routines for 'syncing' multiple Wiimotes work flawlessly — indeed, they work far faster and more reliably than Apple's own Bluetooth tools.
MIDI translators like GlovePIE and OSCulator let you assign any button, accelerometer axis, infrared sensor coordinate or joystick axis to any MIDI note, controller, pressure, program change or pitch-bend command. So, for example, you can use the various buttons to play notes, and use the position or tilt sensors to control volume or pitch–bend. If you are using a synth that responds to MIDI controller commands, you can open and close filters, or change LFO speed or depth. If you send the commands to a sequencing program that has 'MIDI learn' capability, you can control track enabling and disabling, looping, mixing, DSP such as reverb and flanging, and even tempo.
Interactive Control
T
hings really start to get interesting, however, when you process the Wiimote's data through programs you create yourself, using tools like Cycling 74's MaxMSP (for Mac and Windows) or Miller Puckette's platform–independent Pure Data (or 'Pd'). These are flexible object–oriented programming languages that let you build custom MIDI and audio processing and control systems, and interface them with real–world inputs and outputs.
There are two ways to get Wiimote data into these programs: either through MIDI or 'raw'. In the first approach, the output of the Wiimote–to–MIDI translator shows up as an 'instrument' at MaxMSP's or Pd's input. A somewhat more sophisticated approach is to bring the Wiimote data directly into the programming language. Both programs are supported by hundreds of user–created 'objects' that provide connections to external hardware and software, so it's not surprising that someone has built a Max object for the Wiimote.
'aka.wiiremote' is a MaxMSP object that makes all the Wii Remote's data streams available as messages within the MaxMSP environment.'Aka.wiiremote' was created by Japanese multimedia artist and software developer Masayuki Akamatsu, and is available for free. Some interesting aspects of Akamatsu's object are that you can specify the Wiimote's data speed (the default rate is 20 messages per second), which might prove useful if you need to worry about 'MIDI choke', and that it also lets you send data to the Wiimote, turning its LEDs on and off and controlling its internal vibrator (though the musical use for this escapes me). In my experience of using MaxMSP, there's no discernible difference in performance between using raw Wiimote data and having it converted into MIDI data first.
What MaxMSP and Pd allow you to do is effectively re–program the Wiimote's controls on the fly: the buttons and continuous sensors can become interactive with each other, with the action of any control depending on the state of the others. Depending on which buttons are being pushed, for example, the horizontal infra–red tracker might be controlling filter frequency, or wave shape, or reverb time, while the Nunchuck's joystick might at different times handle sample start point, or octave, or flanger depth. As the accompanying box about a piece a student and I created for two Wiimotes illustrates, the number of musical gestures this opens up is huge — so huge, in fact, that the most important job of the composer is to know when to stop making new ones!
The Nunchuk add–on incorporates a further three–dimensional accelerometer.The astonishing amount of technology built into Wiimotes means that they (and whatever Nintendo and their competitors come up with next) will provide unlimited fertile ground for experimental musicians of the future. Such inexpensive 'non–musical' gadgets could completely redefine what musical performances will look like, and sound like. They have the potential to provide a degree of expression beyond even the most expressive of traditional instruments, to say nothing of the crude control provided by computer mice, trackpads, mixers and turntables. You're going to start seeing a lot of musicians using Wiimotes and similar devices to do new and fascinating things on stage. And if you're the pioneering sort, I'll bet you will want to be one of them.
Creating A Performance Piece For Two Wiimotes
One of the challenges of building new musical instruments isn't so much getting them to do things as limiting what they can do. If an instrument has infinite capabilities, it will take an infinite amount of time to learn. So when composition graduate student Phil Acimovic and I put together a piece for duelling Wiimotes for a recent new–music concert at Tufts University, we had to decide on a basic method of playing the Wiimotes, and — despite all the temptations we knew would come our way — stick to it.
One of the MaxMSP objects created for 'Imaginary Dialogues'.The heart of the system was a MacBook Pro equipped with Reason and Max software. The piece was divided into 12 sections, and each of us had a Malström or NN19 module for each section, which were routed through various reverb, flanger, vocoder and distortion modules. We used OSCulator software (see main text) to receive the data from the Wiimotes, and processed it through a set of complex MaxMSP patches. Audio was handled by a MOTU 828, feeding two pairs of Bose L1 line–array speakers.
In the Malström modules, pressing the large button on the front of the Wiimote played a random note, the pitch of which was determined by the horizontal angle of the Wiimote. The notes were restricted to certain scales — major, minor, and pentatonic — which were selected by the three buttons in the middle of the Wiimote, thus giving these sections a sense of tonality. Moving the Wiimote in the vertical direction changed the volume, while tilting the Wiimote panned the sound between the left and right speakers.
Other parameters were changed by pressing one of the four 'directional' buttons or the large 'A' button on the Wiimote, and moving the joystick on the Nunchuk. For example, when the 'Up' button was pressed the joystick controlled pitch–bend; pressing the 'Down' button gave the joystick control over the envelope release time; and pressing the 'A' button meant the joystick adjusted the reverb wet/dry mix.
In the NN19 modules, pressing the large Wiimote button spat out a string of random one–shot samples, with the length determined by how long the button was held in, and the tempo set by the horizontal angle of the Wiimote. The vertical angle controlled pitch–bend, as well as the range of a random LFO assigned to pitch, so the overall effect was that of random pitches in a moving 'cloud' that went from high to low. The directional buttons and joystick were used in the same way as in the Malstrom patches, giving control over volume, LFO rate, filter frequency, envelope release, flanger feedback, reverb mix, vocoder mix and distortion.
Our sounds were all based on the human voice. In Malström we used vocal, choral, and throat–singing samples, and in the NN19s we loaded a wide variety of samples, including Mongolian language lessons, a rabbi chanting in Hebrew, a woman reading James Joyce's Ulysses, and snippets of political speeches from the '40s to the current decade.
When I shook my Nunchuk hard, it sent the message to Max to move to the next section. Phil designed a very clever visual feedback screen so that we could see what we were doing in each section: the screens reminded us which synth parameters were assigned to which controls in the patch we were playing, and also showed us the current values of all of the adjustable parameters.
We called the piece 'Imaginary Dialogues', since it was about two people trading vocal sounds that were mostly devoid of meaning. At the end, as the sounds got louder and nastier, a member of the audience walked up on stage and picked up the computer, closed it, put it down, and walked off. Phil's and my sounds started drifting upward in pitch, until they disppeared into the ultrasonic ozone in a cloud of reverb, and we were silenced.
This ending — which I must say was very effective — was the only part of the piece not controlled by the Wiimotes: it was generated by the laptop itself. For the past three years or so, Apple have included a feature called the 'Sudden Motion Sensor' in their laptops, which parks the internal drive head if you should drop the computer, protecting it (hopefully) against damage. The sensor it uses is very similar to the one in the Wiimote: it's a three–axis 2G accelerometer.
A number of hackers have found a way to tap into the data produced by this feature to play games or to turn a MacBook into a seismograph. In addition to his 'aka.wiiremote' Max object, Masayuki Akamatsu has avaialble an object called 'aka.bookmotion' which sends out the three–dimensional values generated by the accelerometer. We simply looked for those numbers to change, indicating that the computer had been picked up, and when they did, Max executed a subroutine that ended the piece for us.
You can see our performance of 'Imaginary Dialogues' at tuftsemid.com.
The Radio Baton & The Airdrums
Swinging things around in the air to control electronic sound may seem like a new idea, but in fact it's not. Two predecessors of the Wiimote were Max Matthews' Radio Baton and Palmtree Instruments' Airdrums.
Max Matthews, 'father of computer music', with the Radio Baton.Photo: Roger LinnThe Radio Baton was invented by Matthews (considered by many the father of computer music, after whom the Cycling 74 Max software is named) and Robert Boie in 1987. It uses two 'wands' that resemble small gong mallets, within each of which is embedded a low–frequency radio transmitter. You play the instrument by waving the wands over a flat surface containing five copper plates that act as antennas. The signals from the transmitters are compared at the different antennas, and are processed mathematically to determine the position of the batons in three–dimensional space. The Radio Baton was never a commercial product, but several were built and are still in use by composers such as Richard Boulanger and Andrew Schloss.
The Airdrums.Photo: Pat DownesThe Airdrums were the most direct ancestor of the Wiimote. Introduced in 1987, the Airdrums consisted of two cylinders, each about the size of a cucumber, inside of which were three accelerometers responding to vertical, horizontal, and rotational motion. The cylinders were connected to a hardware controller box, which interpreted the accelerometer data and let you turn it into MIDI notes, chords, controllers, patterns, rhythms and tempos, and also let the various dimensions interact with each other. For example, one motion might set up the notes in a chord while another motion would execute it. Programming — deciding which accelerometer in which hand was going to perform which function — was done entirely on the box, using some 54 membrane switches and a two–line LCD.
The Airdrums got stunning reviews in the electronic music press, but never really caught on, for a number of reasons. First, the system was not easy either to program or to learn, and the MIDI community at the time was hungrier for familiar interfaces (keyboards and guitars) that could produce new sounds than for devices that required a whole new mindset to play. Second, the electronics were fairly delicate, which was less than ideal for musicians who wanted to use it on the road. Third, the Airdrums were far from cheap: the list price was $1895 — which, allowing for inflation, would today buy you 100 Wiimotes!