Quartz Composer has always had an easy way to play movies. You drag and drop your movie into the editor, connect the image ports and the movie plays. However, its control over movies has left much to be desired. We can set the timebase to external for a movie, which the exposes the Patch Time parameter. If we change that the movie shuttles to our new position, but we can’t get it playing again without changing our timebase. And you can’t change the timebase with an external controller, so this method is out for live performance.

Also, to play the audio track of a movie, you have to switch to Asynchronous mode. In Leopard, this also meant losing any control over playback.

The movie loader in Snow Leopard changes all of that. We can zoom between positions in our movie, while the video and audio play continuously.

Let’s take a look.

Asynchronous mode in Snow Leopard gives us a lot more options than it did before. Here are the new parameters we’ll focus on:

Reset Signal:

Shuttles the movie back to the Selection Start and continues playing.

Selection Start:

The playhead, effectively. Changing this will not affect current position until Reset = True.

Rate:

Playback speed for audio and video. Slow-mo effect also pitches audio up and down. Works in reverse.

Building it Yourself

In this short tutorial, you’ll learn how to use these parameters to control your video and audio.

1. Start a new composition and add a Clear patch and a Billboard.
2. Set the Billboard’s Width to 2.
3. Drag in a movie from your hard disk (an SSD is highly recommended). You can download a free high-quality countdown movie at the end of this article.
4. Connect the Image output of your Movie Loader to the Image input of your Billboard.
5. Use the Inspector (Command-i) on your Movie Loader to turn on Asynchronous mode.
6. Set the Movie Loader’s Rate to 0 to stop playback.
7. Change the Selection Start to an arbitrary amount and check the Reset box to see your movie jump to the new point. Note that you have to uncheck and then recheck the box to get it to jump again.

Now we just have to combine these actions – setting a new Selection Start and resetting the movie. I’ve got a finished version of this page available for download at the end of this article. I recommend you download it and open it up.

Playing With My Version

In this setup,  we still have the Movie connecting to the Billboard, and we’ve got two new additional patches. We’ve got the Keyboard patch, which sends key signals through the Javascript patch, which interprets them, sends the Position data out to the Selection Start on the Movie, and also sends a Reset signal.

Notice that the Movie’s Duration is being fed into the Javascript patch. This is how the Javascript patch knows where in the movie to jump to. For the curious, here’s the Javascript code:

Using the home row keys on your keyboard (a, s, d, f, j, k, l, ;), you can jump to 8 different spots in the movie. You can also change the Rate for some fun time-bending effects. So there you have it – controlling a movie – With Audio – in the new Quartz Composer for Snow Leopard.

Next time we’ll take a look at swapping out this keyboard patch for a Midi Notes receiver so we can control our videos with Ableton Live.

Coming in early 2010: Realtime Graphics With Quartz Composer

The first how-to book on using this awesome language. Co-authored by Surya Buchwald (Momo the Monster) and Graham Robinson (VJ Shakinda).

For my first foray, I’m working on a Four Layer system.

1. Background – Scene-setting clips.
2. Playground – Video Instruments (Quartz Composer)
3. Foreground – Elements on black, textures.
4. Overlay – Logos, text.

Take Control

I’m a gear-hound. It used to take me HOURS to get ready for a gig as I packed every device, cable, work surface, and display device into my poor old ’85 Corolla. One of the advantages to sussing out a system like this is that I get a chance to refine my physical interface.

Software: VDMX

I chose VDMX for this setup because of its tight integration with Quartz Composer, which is integral to this particular structure. It can read my native QTZ files,  generates GUI elements for published inputs, and can use QC compositions as pass-through FX. Don’t worry if that’s all gobbledy-gook to you for now. Just trust me when I say it’s the bee’s knees for QC integration.

Faders/Knobs: Evolution UC-33

The (now-discontinued) UC-33 packs a metric tonne (or so) of faders and knobs into a reasonably compact, bus-powered interface. Yes, the knobs are a bit close together and the faders certainly aren’t going to win you the DMC – but it will get this jobbie done.

Buttons: Monome 128

The patron saints of Compactness, Versatility, Minimalism and Sustainability had at each other one wine-filled night, and the Monome is the only earthly record. Or it’s just a box of buttons that can light up. Thanks to some beyond-the-call-of-duty programming by Dave Lublin, I was able to use the Monome to fully control my clips. 4×4 groups of buttons control 4×4 grids of clips, with additional controls for switching Media Bins on the fly and scrolling up and down through larger bins. The Monome also acts as a 12-key MIDI keyboard for the Quartz Composer patches that will be looking for one.

Fun Factor: Logitech Gamepad

I love working with a gamepad. It takes two hands, which can be tricky when you’re trading to fade layers in and out. I’m still working on that part. But it’s a barrel of monkeys  to work with (translation: fun), and introduces another interesting structure to work with. Also, it’s hella cheap. No worries about handing one of these to the 5-year-old kid at Maker Faire (just keep yer paws off my Monome).

The Verdict: Oh yeah.

I built this system just in time for the ’08 Decibel Festival. Even though a lot of the programming didn’t come together until the last minute, I had a solid, fun show. It certainly helps that I was playing with an awesome band, but having that structure there really gave me something to focus on. And I’ve got a host of ideas for real 4-layer scenes I want to build. For the sake of the tutorial (and the performance), I just wedged my existing content into this new framework. I think once I’ve had some time to build fresh content for this system, I’m going to have a really solid, extensible show on my hands.

Has anyone else thought through their performances like this?

If you’re not familiar with Lucifer, it’s a VST/AU plugin for realtime beat-based cutup/repeats of audio. If you’re not already a Lucifer user, this article may not interest you much – but I do suggest you check out the video to see what’s possible.

What you’re going to do is route the MIDI from Lucifer out to another program that will do Video cutups. This is useful for more than just video – with the MIDI signals coming out of Lucifer, you can control and trigger and MIDI-capable software and hardware. But if you’re still reading at this point, you probably already knew that.

The first thing you do is set up a new graph in Lucifer. Under the ‘Effects’ tab, select ‘MIDI CCs LAT’ and turn on the effect by clicking the little red button just to the left of the dropdown menu. Now for video use, we want a perfect rising graph in 64th-note intervals. Yes, you’ll have to draw this by hand – but it’s easier if you start with bigger chunks and work your way up to the finer control of the 64th notes. Update: Steve Duda tells me that you can use the ‘shape’ tool to draw the graph automatically. Thanks, Steve!

Then copy-and-paste this graph to all of the other graph sets for which you want MIDI out.

The last step within Lucifer is to set the MIDI Control that will be sent out by doing a click-and-drag in the box just to the right of this MIDI symbol. In my example, I use CC#125.
The last bits you need to do are in Live. Set up a MIDI track, turn its monitor to IN, and set it to receive MIDI FROM whatever track Lucifer is on. If Lucifer is the only MIDI item in that track, it should come up in a little box below the Track Name you’ve selected. If not, the select it, silly.

Now you set the track’s MIDI TO port to send wherever we like. On a Mac, you can use the IAC bus. In this case, I used Isadora Virtual In, which is generated by Isadora whenever it’s running.

This tutorial has covered enough ground without going into a step-by-step Isadora tutorial. I’ll give you the general idea of what I did to get the video working. If you’ve used any node-based languages: Max/MSP, Quartz Composer, Reaktor, Cinema 4D Expresso – Isadora is quite similar, it just has different names for things.
In Isadora, I have a MIDI CC Watcher that routes the value information from Lucifer into a Movie Player, hooked up to a Projector for playback. I also use a Real-Time Watcher to observe the incoming MIDI Clock information from Ableton Live and calculate the Loop Length needed to play the video at 1x speed in time with my loops from Live.

If you want any more info, or some clarification – leave a comment and I’ll see what I can do to satisfy you.

Roadmap:

• Only allow control via Lucifer while a modifier key is held down (otherwise the Audio in Isadora suffers)
• Map Lucifer modifer and gridset keys to Monome.
• Map ‘Reverse’ key in Lucifer to also reverse video in Isadora.
• Your suggestion?

In Leopard, we need to open the Patch Creator. In Tiger, this is always open, and called the Patch Library, found anchored to the left side of the Editor. Type ‘Clear’ into the search field to drill down to the patch we want. Drag-and-drop the Clear patch to the Editor.

Now we’ll do the same with a cube. Type ‘cube’ into the search field, and then drag the patch into the Editor.

Select the Cube and press Apple-i to bring up the Inspector. This is where we can hand-adjust all the aspects of our Cube.

The cube is a little too big to work with, so we’ll take it down in size. Type 0.25 into the Width, Height and Depth fields.

Now let’s play with the X, Y and Z positions of the cube. Spin the dials to the right of the text fields labeled X Position, Y Position and Z Position and watch how the cube moves around in space. To change the values, you need to click and hold on a dial, and the move the mouse in small clockwise circles to raise the value, and counter-clockwise circles to lower it.

Let’s reset these values to 0 by typing directly into the fields.

Now let’s play with the X Rotation property. You’ll notice that the changes we’re making are very small. This is because Rotation is measured in degrees, so to go all the way around once, we need to increment from 0 to 360. Hold down the Shift key for faster rotation. We can recognize that the Cube is three-dimensional, but it’s hard to tell, and not very pleasing to look at. Let’s add some lighting to flesh out the look.

Type ‘lighting’ into the patch search, then drag and drop a Lighting environment into your composition. Notice the square edges on this patch. Our previous two patches, which are both Renderers, have rounded edges. The squared edges of this Lighting Environment patch tell us that we can put other patches inside of it.

Select the cube and press Apple-X to Cut it to your clipboard just like you would with a Word Processor. Double-click on the Lighting patch and press Apple-V to paste your Cube back inside. In Leopard, you’ll see the above breadcrumb trail letting you know that we’re inside the Lighting Patch, which is inside the Root patch of our composition. In Tiger, you may have to click on the ‘Hierarchy Browser’ to see this. Click ‘Edit Parent’ to return to our Root Patch.

Back in our Root Patch, click on Lighting and press Apple-i to bring up the Inspector. Change Attenuation to 0.1 to smooth out the lighting, and play around with the Light 1 XY and Z Position to see how it affects the composition.

Click on the color box next to Light 1 Color to bring up a color wheel. Pick a new color and observe the changes in your viewer.

Next we’re going to bring in an Interpolation patch which will provide some automated control.

In the Inspector for the Interpolation patch, choose a Start and End Value of 0 and 360, respectively. This tells the Interpolator what values we want it to output.

Connect the Result from the Interpolation patch to Y Rotation on Cube. Now the Interpolation patch is counting from 0 to 360 (using decimals in-between for a smooth output) and we’re using those numbers to change the Y Rotation on the cube. Change the Duration of your Interpolation from 1 to 0.1 and watch it go freakishly fast. Change it to 10 and observe it slow down. Revert it to 1 second and let’s move on.

Next we’ll add an Audio Input patch so we can interact with our composition.

Connect the ‘Volume Peak’ output to the ‘Height’ input on the Cube.
In your viewer, your cube should now be responding to your default system input – on a laptop, this will be your internal Mic. On a desktop, you may need to plug in a Mic or some other External Audio source to see it respond.

NOTE: It is possible to redirect iTunes to your QC comp using a utility called Soundflower – look for a tutorial on this topic soon.

Looking at the viewer, you’ll notice that our Cube is not hitting the top and bottom of the display area. In order to change this, we’ll add a Math patch. We’ll route the signal through the Math patch (as pictured above) before sending it along to the cube.

In the Math Patch Inspector, we’ll change the Operation to Multiply, and try ’4′ for the amount. In Tiger you will default to having two operations – you can leave the other one blank.

Now we should see the cube stretch all the way to the top and bottom of the screen. We can play with the multiplication amount until we find a spot we like.

Now it’s time to experiment on your own. Open up patches and change some values, see what happens. When you find a change you like, hook up an Interpolation or Audio Input patch to it. Stick Math patches in your routing streams to take control of your values. Experimenting like this works to cement the ideas you learn, and steadily increase your understanding of the Quartz Composer System.

So for this first part, I had to find something related to ‘water’ to videotape. Thankfully, there’s a fountain right near my office (the one at the Grove for any Los Angeles readers). I always carry my little camera around in my man-bag, so on the way to work one morning I stopped by the fountain and videotaped it like all the other tourists.

After work, I transferred the vids over to my laptop, brought them into Final Cut and started cutting them up to short clips – I specifically looked for ones that had rather defined movement, which would help in matching them up with audio.

Next, I decided to spice up my clips a little bit. The colors on my lil camera are alright, but nothing to write home about. I fired up some color-correction tools and made the water a bit more aqua to help it pop on-screen. Then I used a combination of the contrast tool and the trusty ol’ Kaleidescope to abstract the clips a bit.

Since I took the video in a public place, I knew on-location audio recording was out. So once I had a nice selection of clip possibilities, I opened up Ableton Live.

In Ableton, I fooled around with a free subtractive synth I’d found called Automat. Once I had a sound I liked, I made up a few patterns, and then started flipping back and forth between Final Cut and Ableton live to see what might match up. When I found a pair that looked close, I would fine-tune it a bit in Ableton, and then export it to Final Cut. Rinse and Repeat.

After about an hour, I had sounds for all my Final Cut clips. I lined them up, trimmed the clips to match the sound lengths (making sure that the first frame of video started on the sound) and set up a Batch Export.

My specs:
640×480
Motion-JPEG B at about 85% Quality
Linear PCM Audio

Once done, I uploaded them to archive.org and compiled all the clips into the AV Challenge #2 Clip Packs.

In Part Two (coming soon) – I’ll load the clips up and start making beats and melodies!

This is a tutorial on how to mix video using a Nintendo Wii controller and VJ Software such as Resolume or Isadora for your PC. I used a Dell Inspiron 8100 with a gig of ram running WinBlows XP, some kinda ATI video card that has disappointed me in the past, and a 2.4 MHZ processor. Running Vista? Haven’t tried it yet. If you have Vista and this walkthrough works for you, let me know.

About the Wii (from the excellent wii to midi walkthru by ANTIVJ.com)
In addition to the traditional buttons, force feedback and the integrated mini-speaker, the true innovation is the presence of a gravity and acceleration sensor (the ADXL330, used in the army or automobile safety, for airbags systems) inside the Wiimote which collects the movements and acceleration of the Wiimote on each of the 3 axis, X, Y et Z.The good news is that rather than using a proprietary protocol to transmit the infos from the Wiimote to the Wii, and then lock its hardware to keep an exclusive use of it, Nintendo has made the excellent decision (probably by commercial will) to use the Bluetooth protocol, standard of communication on PC and Mac for several years. This means that you can connect your Wiimote to your computer, and program it rather simply for any use you want.

About this tutorial
Use this tutorial however you want, feel free to add to it, be sure to forward it to whomever and give it away for free. If you charge some lazy bastard for this tutorial, you suck. If you use it for a performance and you make money– you rock!

What you will need
- A Wiimote, of course (approximately $35 online or at your local video games shop).
- A Bluetooth Dongle (if your computer doesn’t have Bluetooth already, about $30).
- Bluesoleil (shareware, it usually comes for free with Bluetooth dongles).
- GlovePie (freeware). Converts the signals sent from your wiimote to Midi (or anything else).
- MidiOX (freeware). Manages MIDI. Although I can use it to route my in and outputs, I really only use it as a MIDI monitor. You can contribute to MIDIOX thru Amazon.com– I gave them 10 bucks.

Other resources:
Using the wiimote with Resolume (very excellent walkthru, complete with midi scripts for RSLume):
http://www.resolume.com/forum/viewthread.php?tid=3102
Wiimote to Midi tutorial from AntiVJ:
http://crustea.vjfrance.com/article-130714.html

STEPS:
1) Connect your wiimote to your pc
2) Convert the signals the wiimote sends to MIDI
3) Rock it, rockstar!

STEP ONE: Connect your wiimote to the PC

If you have Bluetooth built into your computer skip to part 2. If you dont, you may take this opportunity to install a ridiculously expensive Bluetooth adapter and construct a delicate web of buggy freeware– PC only, of course.

A. Install your Bluetooth USB adapter (I used a $30 Kensington dongle from Best Buy). I had to install the drivers for a minute, plug the thing in, etc. Eventually you should get that little message that says “your hardware is ready to use”. If not, re-install, unplug it then plug it back in, go to the manufacturer’s website and try to install the drivers there, make sure your adapter is compatible with your PC and with a wiimote– you know the drill. I have heard of people using Bluesoleil software to access their Bluetooth, but I always like to use the software that comes with the device, and I thought Bluesoleil was kinda buggy.

B. Configure your Bluetooth adapter. Once your driver and the Bluetooth adapter is installed, you should see the funky blue Bluetooth logo in your toolbar. Right click on it (or go thru your Start menu) and do a “Explore My Bluetooth Places”. A folder will open up, click on “Bluetooth Setup Wizard”, and you will get this dialog box.Click on the second radio button to “find a specific Bluetooth device and configure how this computer will use its services”. Hit the 1 and 2 buttons simutaneously on your wiimote (switches it to “discoverable mode”) then hit “next” in the dialog box. Watch the flashing blue lights while the wiimote tries to talk to your Bluetooth adapter!

C. Select your Bluetooth device (in this case, a wiimote). Now the dialog box will say “Select a Device” and there will (hopefully) be a picture of your wiimote to select. Select it, then hit “Next!” If your wiimote does not show up, try to re-initiate: hit “search again” in the dialog box, switch to “discoverable mode” on your wiimote (hit the 1 and 2 buttons simultaneously), etc.Once you select your wiimote and hit “next”, you will get to the “pairing” screen. Ignore it– hit “skip pairing”. I tried to initiate pairing, but I couldn’t find a number on my wiimote, except some weird serial number that didn’t seem to do anything. What is pairing? I don’t know. This is taking too long.

D. Configure your wiimote. Now that your computer sees your wiimote, you need to select it. Again. It seems like I needed to constantly switch to “discoverable mode” on my wiimote (hit the 1 and 2 buttons simultaneously) during this entire process.Hit “Finish”. Your Bluetooth logo in your toolbar should turn green. Your wiimote is now connected to your PC!At one point, when installing on a Dell Inspirion5100 this step did not work– I went back, uninstalled, re-installed, and it worked fine. Also, these type of networky softwares seem to work poorly with P2P software such as Limewire and Bittorrent- be warned!

STEP TWO: Convert the signals the wiimote sends to MIDI

Now that your wiimote is talking to your computer, it’s time to configure some softwares to listen to what it says! You will be using GlovePie, a program that was originally created to work with the Essential Reality P5 Glove– classic! This software developed by Carl Kenner has the role of converting any type of signal (from a mouse, Joystick, a P5 glove) into another signal (MIDI, variables…) using a simple scripting language and/or a GUI.

A) Start up GlovePie. I don’t recall if it even needs an install, but I remember having some problems when running it off my desktop, then moving it to my Programs folder, then running it again.Anyway, when you open it up it displays an empty script editor, basically you type in the commands you want your control module (in this case, the wiimote) to execute and hit “Run”. There’s an autogenerate, but I have not really experimented with it yet.Raw coding is intimidating, but fortunately there are loads of scripts provided with the install: for example, go to the “File” dropdown menu, then open “WiimoteScripts/WiiMouse 0.1.PIE”, and click on RUN, and you can now control your mouse cursor with your Wiimote.This is what I used to make sure that the wiimote is actually working– up untill now, the only proof you have that it works is a stupid dialog box. If you run “WiiMouse 0.1.PIE” and your cursor drifts up to the top left of your screen– your wiimote has not been detected properly. Go back to some previous steps, try again! I sometimes use “WiimoteIdentifier.PIE” to check if it sees any wiimotes, but it seems like it can “see” the wiimote without “hearing” the data. If you are dancing your mouse around the screen with your wiimote, you are almost done!

B) Load the Wiimote to Midi script into GlovePie. Antivj.com has the newest version of the script here, or you can copy, paste and RUN the script below. After you copy and paste this code into GlovePie, be sure to save it for later– call it something like “WIItoMIDI.PIE”. Be sure to uncomment/comment out the appropriate sections to disable key commands from your wiimote if you are planning to use it more… unconventionally. This script ( beta0.1), you get the value of X, Y and Z slope value from your Wiimote, which are converted into #CC values 30, 31 and 32, variables from 0 to 127. Sometimes this seems to make sense to me, sometimes not. Are we done yet?
/********************************
Wiimote to midi, beta 0.1
Use your Wiimote to control Ableton live, Arkaos midi..
contact: crustea@gmail.com
www.antivj.com
********************************/
/********************************
Wiimote’s buttons
********************************///Keyboard notes
//To send “keyboard events” instead of “midi notes”, uncomment this part, and comment “Midi notes” part above.
//Values before the “=” caracter are the keyboard events that you emulate./*
Up = Wiimote.Up or Wiimote.Classic.Up
Down = Wiimote.Down or Wiimote.Classic.Down
Left = Wiimote.Left or Wiimote.Classic.Left
Right = Wiimote.Right or Wiimote.Classic.Right
Q = Wiimote.A or Wiimote.Classic.a
W = Wiimote.B or Wiimote.Classic.bOne = Wiimote.One
Two = Wiimote.Two
Home = Wiimote.Home or Wiimote.Classic.Home
Minus = Wiimote.Minus or Wiimote.Classic.Minus
NumPadPlus = Wiimote.Plus or Wiimote.Classic.PlusX = Wiimote.Classic.x
Y = Wiimote.Classic.y
v = Wiimote.Nunchuk.ZButton or Wiimote.Classic.ZL or Wiimote.Classic.ZR
L = Wiimote.Classic.L
R = Wiimote.Classic.R
C = Wiimote.Nunchuk.CButton
*/

//Midi notes
//To send MIDI notes, uncomment this part (done by default), and comment the “Keyboard notes” part
//The value before the “=” caracter is the midi note.

midi2.C0 = Wiimote.Up or Wiimote.Classic.Up
midi2.D0 = Wiimote.Left or Wiimote.Classic.Left
midi2.E0 = Wiimote.Right or Wiimote.Classic.Right
midi2.F0 = Wiimote.Down or Wiimote.Classic.Down

midi2.G0 = Wiimote.A or Wiimote.Classic.a
midi2.A0 = Wiimote.B or Wiimote.Classic.b

midi2.B0 = Wiimote.Minus or Wiimote.Classic.Minus
midi2.C1 = Wiimote.Home or Wiimote.Classic.Home
midi2.D1 = Wiimote.Plus or Wiimote.Classic.Plus

midi2.E1 = Wiimote.One
midi2.F1 = Wiimote.Two
//midi2.A2 = Wiimote.Nunchuk.CButton
//midi2.B2 = Wiimote.Classic.x
//midi2.C2 = Wiimote.Classic.y
//midi2.D2 = Wiimote.Nunchuk.ZButton or Wiimote.Classic.ZL or Wiimote.Classic.ZR
//midi2.E2 = Wiimote.Classic.L
//midi2.F2 = Wiimote.Classic.R

/********************************
Setup for gravity values
********************************/

// these are the magic line, which converts X, Y and Z rotation values to Midi #CC 31, 32 and 33.
var.xOffset = 1
var.yOffset = -28
var.zOffset = 2

var.xRot = Wiimote.RawForceX + var.xOffset
var.yRot = Wiimote.RawForceY + var.yOffset
var.zRot = Wiimote.RawForceZ + var.zOffset

midi2.Control31= (EnsureMapRange(var.xRot, 26,-26, 0,1))
midi2.Control32= (EnsureMapRange(var.yRot, 26,-26, 0,1))
midi2.Control33= (EnsureMapRange(var.zRot, 26,-26, 0,1))

//Old code: (MapRange(var.xRot, 26,-26, 1,127)-1)/127
/********************************
Setup for acceleration values
********************************/
//Just a test for now, comment the line above to disable X acceleration to midi note
midi2.C3 = Wiimote.RelAccX >= 15

C) LAST STEP! Now it is time to check to make sure Glovepie is sending the appropriate data by installing and running MidiOX (a virtual MIDI port, which emulates a MIDI device to use like a “bridge” with your midi softwares).After you install MidiOX, you should see something like this. The blue numbers should be scrolling as you move the wiimote, and green numbers should appear when you click on some of the buttons. If this is not the case, hit “Select MIDI devices to open” (the round blue button that looks like a midi input). I just opened and closed midi ins and outs untill it worked. For more info, check out the MidiOX site.If the numbers are scrolling rapidly, YOU ARE GOOD TO GO! Your wiimote is going into GlovePie, and then into MIDIOX, and from there you can send it to another MIDI device, or use it to control video mixing software.I used the Wiimote as a motion sensor for an interactive golf game in a freight container (http://www.angelsgateart.org/events/fr8.html)My next step it to apply this MIDI input into Resolume and Isadora, and do more conventional video mixing using the motion controls as opacity toggles. Wish me Luck!

The Playstation 2 Game Controller is a wonderful interface. I debuted this installation at the 2007 Maker Faire, and found that kids and adults alike (but especially kids and 20-something gamers) were able to understand and control this interactive Audio Visual System very quickly.

This system builds upon the work I started here: A/V Synth Controlled by Game Pad, and the graphics were inspired by the Interface Design of Battlestar Galactica. Watch the video above and come back for Nitty-Gritty technical details.

First up is Junxion – software that converts USB to MIDI messages. First, a disclaimer – I bought this back when it was $29, and I think it’s a bit overpriced now (€25 for the basic version and €75 for the deluxe). You can try arj.MultiControl – the last version I used was buggy, but they may have fixed things up since then. In any case – here’s what my Junxion setup looks like:

I have most of the buttons sending CC messages, with the exception of the four circular buttons on the right side of the controller (this would be Triangle, Circle, X, Square on a PS2), which send notes, since they drive the synthesizer. This midi gets sent to my Ableton Live setup:
Track 1 are my two drum clips – I trigger the Play Clip with L1 and R1 and the Track Stop with either L2 or R2. There’s a ‘Stereo Beat Cutter’ Effect – I control the ‘Chance’ parameter of this with my Directional Pad, defaulting to ’0′ when I’m not pressing anything, and jumping to four possible ranges when I press Up, Down, Left or Right.

Track 2 is my Synth. I pitch the notes down 2 octaves and run them through an Arpeggiator to the Joystick preset. The Analog Sticks on the gamepad control ’2nd Pitch’ and ‘Filter’ in the Joystick Synth.

Track 3 is my MIDI-OUT track. Here I take the Arpeggiated MIDI generated by Track 2, and send it out to my IAC Port (the built-in MIDI routing device in the Mac OS). This brings us to:

The Quartz Composer patch to run the visuals. Basically, we have a lot of sprites controlled by LFOs, and the controller (for which you can see my handy Macro Patch on the left here) modifies the LFOs, or changes the Alpha of the sprites to make them appear and disappear. One of the tricks I use is this patch to the right, called ‘MIDI Keys 12′. This was a simple enough idea, pointed out to me by Electronic Music Producer Chiral when I was brainstorming for ways to improve a synthesizer-based patch. Inside is a giant mess, of which you can see a tidbit below.

Basically, I take every possible octave for each note, and run it through a Javascript patch which averages the data from the 10 octaves coming in, and outputs one signal. The end result is a signal that responds to C# no matter which octave you’re playing.

Let me know what you think of this project so far – this is only the first version, and I’ve much more work to do. Ideas I have for the future:

• Left Analog stick should control the tunnel flight, rather than the Directional Pad
• Something should happen when you press Start or Select.
• Third Drum loop? (I could use L2 for a loop, and have R2 be the only ‘Stop’ button)
• The analog sticks can be pressed in for an additional two button clicks – I should use these
• Add multiple controllers – one for drums, one for lead, one for Vox, etc. that all control the same graphic system.
• Your Idea Here.

Downloads

Disclaimer: This is not ‘ready for the public’ at all yet. I’m providing the source code for those who tinker with MIDI routing and interactive Audio/Video. One of my goals is to build a system that will be easy for everyone to download and use, but we’re nowhere near that point.

Ableton Live Project: playsonic_ableton.zip
Junxion Preset: logitech_gamepad.jxn
Quartz Composer Project: playsonicv1.qtz
All 3 of the above in 1 file: playsonicv1.zip

YouTube version: Playsonic:  AV Fun with a Game Controller

• Ableton Live – you’ve probably heard of this one. It’s become a favorite among DJs and Live Electronic Musicians for the way it’s geared towards doing things on-the-fly, with much of the power of a traditional DAW (Digital Audio Workstation). I use it to sequence MIDI and affect audio.
 
• VDMX5 – this is my video performance program. This new version is in Beta, and I’ve found it to have the best AV Triggering performance of any program on the Mac. Modul8 takes a close second place. In this Beta form, there’s some missing functionality and an interface that takes some getting used to – but it works great for this project.
 
• Monome 40h – my controller interface. I first saw this at the Maker Faire in 2006 and was immediately attracted to it for just this sort of purpose. It’s an 8×8 grid of led-lit rubber buttons, powered over USB, speaks serial, OSC and MIDI, and it’s ready to do whatever you can program into it. No proprietary, undocumented secret codes necessary for communication (*cough* MACKIE *cough*) – heck, they even give out the firmware code and the parts listing.
 
• Soundflower – internal audio routing for the Mac. This enables me to send the audio from VDMX to Ableton Live for processing before it is played out through my speakers.
 

To set up for this performance, I first run the driver for the Monome. Next, I run a Max/MSP patch that converts the Monome’s button-presses into MIDI notes, and listens for those same notes coming back, lighting up the corresponding button when it hears them. I set my input and output to the virtual “Max/MSP runtime 1″ port, and the channels to “16″.
Next, I start up Live and my accompanying .als file [monome_sequence_live.als].

I arm my left-side track to receive MIDI from the Monome by pressing the corresponding ‘arm’ button on the interface. Now when I press the buttons on the Monome, I’ll see them light up underneath my fingertips, since I have Live route MIDI from Track 1 (Left-Side Monome In) to Track 5 (Monome Out).


Now that my MIDI control is up and running, I start VDMX and load the Preset that I’ve built specifically for this performance. It consists of two Media Bins looking at the same folder of clips. The bin on the left will listen for the 4×4 grid of notes generated on the left-hand side of the Monome, and vice-versa for the right. Each bin triggers to its own layer, which has been resized to half-size, and moved to the left or right in order to give me the split-screen effect. Now when I push the buttons, I see the video play, and hear the audio.

The audio is actually running through Soundflower (see above), which I have chosen as my default Audio Output in my System Preferences (VDMX does not yet support picking your output from within the program). Live is set to monitor the output coming from Soundflower and run it through my fx chain in Track 7 before sending to Master Output.

I have three empty clips on Tracks 1 and 2 that I use for recording/looping my live input. By pushing the corresponding track button on the Monome, I can start the clip looping and rehearse what I want to lay down. At this point, I’ll either turn on my metronome or a drumbeat to play along with. Once I’m happy with my rehearsal bit, I turn on Overwrite (OVR) using the Monome, and lay down my triggers. I’ll turn Overwrite off again as soon as I’m done so I don’t accidentally record the next rehearsal bit. Sometimes I trigger things a little too late, in which case I’ll simply drag the offending note backwards a little bit in the Live software interface.

.

At this point, I can continue laying down tracks – up to three on each side for now. I can turn on my audio effects for extra oomph.

TO DO: It would be great to have an ‘Undo’ so that I don’t have to mess with the mouse to delete a bad performance. There’s a way to trigger Undo via MIDI by telling Live that my Monome 40h is a Mackie Control Surface, but I can’t pick to enable *just* the undo command – it makes many of my buttons do things that I don’t want them to do (arm/disarm/play tracks, etc.). I’d like to be able to nudge notes without using the mouse as well, though I haven’t figured that one out at all yet.

Comments/suggestions greatly appreciated! I’ll have this setup (or something like it) at the VJ Kung Fu Booth at the Maker Faire this year – you should should come and play.

On the bottom of the NuVJ hardware, you find the ‘Fader Start’ button. Once this is switched on, you can trigger clips by moving the crossfader from one side to the other. This allows you to get your clips ready a second or two before you need them, and trigger them on the beat.

These clips were all compressed as Quicktime MOVs using the Motion JPEG-B codec at about 80% quality. The audio is Linear PCM. I recorded the output directly to an Archos AV700 (like a portable Tivo) and then synced it up with footage from my camera in Final Cut Pro.

You’ve got two Decks – A on the left, B on the right. The one in the middle is your preview, this is what’s going out to your audience. Apple-A (or Ctrl-A on a Windows machine) will take you Full-Screen – if you’ve only got one monitor, you won’t be able to see your interface anymore. Press Apple-A (or Ctrl-A) to go back.

Load clips by dragging-and-dropping onto the 3-by-3 cells in each deck. Take a look at the numbers above the decks – you’ll probably have a 1 on the left and a 2 on the right. There’s a knob marked ‘Bank’ that allows you to switch between 16 sets of 9 clips – for a total of 144 clips you can load up at once.

Trigger clips by pressing the buttons in the 3-by-3 grid on the hardware interface. They light up when they’re playing so you know which ones are currently going. Use the crossfader at the bottom of the unit to mix between your two currently-playing clips. The up-and-down sliders control playback speed – you can speed up, slow down and reverse the clips that are playing.

The jog wheels on the lower end of the hardware affect your currently-playing clip on each side, shuttling them back and forth. For precise control, hold down ‘Scratch’ to freeze your clip, and then move the jog wheel around.

Switch to the ‘Sources’ tab in the upper-left hand corner or the NuVJ software and you can drag in dynamic video – like your iSight, webcam and Firewire cameras, Shockwave Flash files, and on-the-fly text generators.

Once you’ve got a setup you like, make sure to go to File > Save As… in your menu bar and save it to your hard drive so you can easily recall it in the future.

There’s more features to the NuVJ, but that’s plenty to get started with. Put on your favorite music, load up those banks, and have some fun!

PS Let me know if there’s any specific video demos you’d like to see with the NuVJ.