Playsonic: AV Fun with a Game Controller

The Playstation 2 Game Controller is a wonderful interface. I debuted this installation at the 2007 Maker Faire, and found that kids and adults alike (but especially kids and 20-something gamers) were able to understand and control this interactive Audio Visual System very quickly.

This system builds upon the work I started here: A/V Synth Controlled by Game Pad, and the graphics were inspired by the Interface Design of Battlestar Galactica. Watch the video above and come back for Nitty-Gritty technical details.

First up is Junxion – software that converts USB to MIDI messages. First, a disclaimer – I bought this back when it was $29, and I think it’s a bit overpriced now (€25 for the basic version and €75 for the deluxe). You can try arj.MultiControl – the last version I used was buggy, but they may have fixed things up since then. In any case – here’s what my Junxion setup looks like:

I have most of the buttons sending CC messages, with the exception of the four circular buttons on the right side of the controller (this would be Triangle, Circle, X, Square on a PS2), which send notes, since they drive the synthesizer. This midi gets sent to my Ableton Live setup:
Track 1 are my two drum clips – I trigger the Play Clip with L1 and R1 and the Track Stop with either L2 or R2. There’s a ‘Stereo Beat Cutter’ Effect – I control the ‘Chance’ parameter of this with my Directional Pad, defaulting to ‘0′ when I’m not pressing anything, and jumping to four possible ranges when I press Up, Down, Left or Right.

Track 2 is my Synth. I pitch the notes down 2 octaves and run them through an Arpeggiator to the Joystick preset. The Analog Sticks on the gamepad control ‘2nd Pitch’ and ‘Filter’ in the Joystick Synth.

Track 3 is my MIDI-OUT track. Here I take the Arpeggiated MIDI generated by Track 2, and send it out to my IAC Port (the built-in MIDI routing device in the Mac OS). This brings us to:

The Quartz Composer patch to run the visuals. Basically, we have a lot of sprites controlled by LFOs, and the controller (for which you can see my handy Macro Patch on the left here) modifies the LFOs, or changes the Alpha of the sprites to make them appear and disappear. One of the tricks I use is this patch to the right, called ‘MIDI Keys 12′. This was a simple enough idea, pointed out to me by Electronic Music Producer Chiral when I was brainstorming for ways to improve a synthesizer-based patch. Inside is a giant mess, of which you can see a tidbit below.

Basically, I take every possible octave for each note, and run it through a Javascript patch which averages the data from the 10 octaves coming in, and outputs one signal. The end result is a signal that responds to C# no matter which octave you’re playing.

Let me know what you think of this project so far – this is only the first version, and I’ve much more work to do. Ideas I have for the future:

• Left Analog stick should control the tunnel flight, rather than the Directional Pad
• Something should happen when you press Start or Select.
• Third Drum loop? (I could use L2 for a loop, and have R2 be the only ‘Stop’ button)
• The analog sticks can be pressed in for an additional two button clicks – I should use these
• Add multiple controllers – one for drums, one for lead, one for Vox, etc. that all control the same graphic system.
• Your Idea Here.

Downloads

Disclaimer: This is not ‘ready for the public’ at all yet. I’m providing the source code for those who tinker with MIDI routing and interactive Audio/Video. One of my goals is to build a system that will be easy for everyone to download and use, but we’re nowhere near that point.

Ableton Live Project: playsonic_ableton.zip
Junxion Preset: logitech_gamepad.jxn
Quartz Composer Project: playsonicv1.qtz
All 3 of the above in 1 file: playsonicv1.zip

YouTube version: Playsonic:  AV Fun with a Game Controller