Apple’s iPhone — and the significantly more affordable, doesn’t-have-to-be-a-phone iPod Touch — are essentially pocket-sized, intelligent multi-touch controllers. Hooking them up to visual software as controllers simply requires some app on the phone to transmit data, and some way of dealing with that data on the computer side. We’ve already seen this a bit on Create Digital Motion, and we’ve been covering some of the specifics of parsing data with Pd (Pure Data), the open-source, tri-platform patching software, on Create Digital Music this week.

Here’s the basic setup:

On Your iPod/iPhone

You have two options of software to use on your iThing. (You’ll need to “jailbreak” your device, as these are not — and may never be, for all I know — approved Apple apps.)

1. mrmr by Eric Redlinger of Brooklyn (top right):  open-source, editable control screens (requires Mac-only software to edit). See our interview with Eric, including some examples with Quartz Composer.

2. akaRemote.app by Masayuki Akamatsu of Japan: not open-source, not editable, but comes with a set of useful control templates, and you can transmit data to the app. See our look at a recent release. Upcoming Mac-only visualist app 3L has its own special akaRemote-based bridge called i3L, which also runs on iPhone/iPod Touch; see our look at i3L with artificial eyes.

On Your Computer

While the iPhone and iPod Touch have Apple logos on them, all of these apps send OpenSoundControl data. That means any OSC-compatible software will work, which is gradually including more visual software, as well as modular apps like Quartz Composer, Max/MSP/Jitter, Pd/GEM, and vvvv. (I love saying that last one … vvvvvvvvvvvvvvvvvvv. Okay, moving on.)

Of all of these, Pd is the one solution that’s free, open source, and runs on any platform. That means it’s also a viable candidate for translating incoming OSC data to more broadly-compatible MIDI. (i3L has you covered, as it uses a MIDI bridge.)

We have not one but two sets of tutorials / example patches for working with Pd on Create Digital Music, using a patch like Cesare’s, pictured above:

Control Music and Visuals with iPhone/iPod, Free Via Pd

Tutorial: More iPhone/iPod Touch Control With Open-Source Pure Data

So, Is It Worth It?

I usually don’t ask that question, preferring instead to report on what other folks are doing. But it is always worth asking yourself — and it is an entirely personal question. I’m not totally convinced in the case of these devices that I’d want to buy one solely for VJing, but then, what makes this so cool is that it adds on additional functionality to a device. (Too bad Apple is being so uptight about third-party development, but at least there’s an SDK — and plenty of hackers ready to break Apple’s rules.)

My own preference remains squarely with tangible controllers and tactile feedback, especially as some of the advantages of multi-touch are diminished by the iPod/iPhone’s diminutive size. But I absolutely see the argument for using these. What do you think, dear readers?

Gavin Morris has been working on an audiovisual setup with Ableton Live and Isadora, a tasty combination for any Windows or Mac user. Isadora, for those of you who don’t know, is a visually-focused modular patching tool. It’s similar to tools like Max/MSP/Jitter, but by emphasizing the practical needs of visual performance, it’s unusually usable when putting together real-world gigs. Its use by A/V dance troupe Troika Ranch (co-founder Mark Coniglio is also the tool’s creator) has also popularized it in modern dance circles.

Gavin has two tutorials for us to start. The first syncs up Live and Isadora, along the lines we ran here using Live by momo the monster:

AV Cutup Secrets: Using Lucifer & Live

Gavin writes:

It’s similar to Momo’s recent Tutorial but uses a free tool for the VST (Pluggo) and allows control from the Live interface (as opposed to within the VST) This allows you a lot more flexibility and means you can use Follow Actions, adjust loop lengths/positions in realtime and even create a slicer. It is Live>Isadora via OSC but could equally be to many other softwares and could equally use MIDI.

I’ve written a VST to go in slicer channels tool.

Sync Ableton Live to Isadora using a Pluggo VST from digital funfair on Vimeo.

Gavin warns us that the video may "put us off." At first I thought that meant it was NSFW or something, but … well, that’s not the problem. You’ll see. I leave it up to you to decide how you feel about it.

The second tutorial gives you the power of Emergency Broadcast Network-style A/V slicing:

I’ve done a tutorial for a Video Slicer - synching up Live’s slicer to Isadora - same technique but a bit of maths to convert the midi notes Live creates to video position. You can make some quite glitchy s***!

AV Slicer Tutorial - Ableton 7 Slicer with Isadora from digital funfair on Vimeo.

Lots more information at Gavin’s site, Boredbrands Digital Funfair.

He needs someone to build the Mac plug-in, so Max users, if you’re game, go for it!

AV Sync Tutorial

AV Slicer Tutorial

Good as this is, I hope we see some audiovisual setups that work with more asynchronous relationships between music and motion — I know my own tastes for my personal work tend in the abstract. Maybe I’ll have to put my money where my mouth is and write it up myself.

More reasons you love OSC? You can get input from a SpaceNavigator 3D mouse, and translate the data into Max/MSP/Jitter, Quartz Composer, VDMX5 (as mentioned earlier today), Processing, and other tools — plus everything else, via MIDI. The solution: the inexpensive Mac utility OSCulator, covered this evening over on CDMusic:

OSCulator for Mac: Alternative Control, Now with 3D Mice, AppleScript, Combos

3D mice, eh? Suddenly navigating those 3D generative graphics built in Processing for a live performance starts to sound more appealing, huh?

Earlier today:

VDMX5, Now with OpenSoundControl - Everyone Else, You Listening?

So, yeah, we will have to do that “dummie’s guide” to OSC as promised earlier!

VDMX, the “realtime video studio” from the makers of Grid, semi-modular live visualist tool for Mac, and very possibly the “world’s most mature beta software”, continues to pack goodness into each new beta release. But the most recent addition, just now available as part of the primary public beta release, warms my heart. The latest beta adds support for OpenSoundControl or OSC, a protocol for communicating between computers, hardware, and software. With high-resolution data, an open-ended naming scheme, and network savvy, OSC is just plain better than MIDI at at least some tasks. But it’s suffered from a lack of documentation, a vague specification that’s a bit too open-ended and daunting to developers, and most of all, a lack of critical mass as far as tools. One small step at a time, that’s changing.

So far, the VDMX developers have tested their implementation with the multi-touch Lemur controller and inter-app data with Apple’s free Quartz Composer. But OSC is also supported in Processing, Max/MSP/Jitter, Pure Data, and free controller implementations for the Wii remote and Wacom tablet, among many other things. It’s also supported natively by the Monome controller, which is quite a lot cheaper than a Lemur. Want to pipe data from a Wacom into Quartz Composer? Sync data between a Mac running VDMX and a PC running vvvv? Something … uh … else that I haven’t thought of? Quite a lot is possible.

And an iPhone implementation could come next. (Or you can send OSC from iPhone right now, with MrMr.)

You can try this out in the beta that’s available now, though David Lublin from Vidvox warns us:

…there’s still more to do with OSC, currently no way to trigger files, it’s just control data, but the core functionality for sending and receiving is there, so we really just need to start building on top of that for whatever protocols there are to support

…the OSC support is still very much in it’s early stages.. it is there and stable, but we are barely scratching the surface of what we can do with it

Of course, that description could apply to OSC in general, which is why the two CDM’s (createdigitalmusic and createdigitalmotion) will be working over the course of 2008 to work with users and developers to share knowledge and get better implementations in everyone’s hands sooner. It’s something I hope to check in on while I’m at NAMM next week. Stay tuned.

Vidvox + VDMX5

VDMX5 public beta direct download link (Mac only)

Introduction to OpenSoundControl [opensoundcontrol.org]

Speaking of multi-touch OSC implementations, we hope to catch up with CyberPatrolUnit, who’s working on a VJ interface on Lemur, as pictured below. He’ll be bringing it to the CDM NAMM party, so if you’re in LA, come see us next Friday!

Musicians have thousands of years of history when it comes to interfaces and instruments, but visuals are relatively new. Little wonder, then, that visualists are eager to try new interfaces to help make visuals akin to performance instruments. Or, in less lofty terms, let’s get Wii remote wagging in the club tonight.

Over on createdigitalmusic.com, we’re celebrating Game Day — basically, I’m squeezing as many game-related posts into 24 hours, because a whole bunch of tips came in at once. A couple of Wii-related controller solutions jumped out.

Wii + Flash

Getting hardware control in Adobe Flash requires some work: basically, you need a client-server model. The good news is, there are already two Wii-specific solutions out there.

MoteDaemon, Mac OS X
WiiFlash.org, Windows Blog, Download, Google Code

Pretty soon, people are cooking up Minority Report demos with Flash (using Papervision for 2.5D-style 3D in Flash’s 2D world, and Open Dynamics Engine for physics):

I’d loved to see this coupled with something like Onyx for an all-Wii, all-Flash performance app. With Flex and AIR (Adobe Integrated Runtime), this could be the basis for some really hard-core, full-blown apps … though you will be limited by Flash’s slower performance, at least in comparison to C/C++-based tools like Max or even Java.

Looks like Linux users are presently out of luck, unless I’m missing something (feel free to chime in if you know a cross-platform alternative).

Wii + MIDI, OpenSoundControl (OSC)

For a more app-agnostic solution, you can hook up a Wii to send MIDI or, ideally, OpenSoundControl (OSC) to apps that support it (vvvv, Max, Pd, and hopefully VJ apps soon — I’ve heard a couple of developers working on it).

On Mac, you can send both OSC and MIDI with one app, perfectly-suited to taking data from the Wii controller:
OSCulator. OSCulator is a great tool for the Wii, but it also shows promise of what a hardware input hub could generally look like, with open-ended inputs controlling visuals and sound rather than pre-defined, MIDI-style keyboards and knob and faderboxes as have traditionally been used in music.

On PC, GlovePIE is an awesomely-powerful scripting tool for use with Wii and other game devices. I’ve talked about it endlessly before, but I’ll stay quiet — just go. Get it. Enjoy. In fact, with OSCulator on Mac and GlovePIE on Windows, it’s hard not to be insanely happy on either platform.

Another interesting out-of-the-box alternative, though, is the new Wiinstrument. It’s largely geared for playing back drum samples, but it’s not hard to take that metaphor and use those control changes and other MIDI messages as visual controls — especially if you think of your “drum kit” as sets of visual clips (video or other visuals).

Wiinstrument on createdigitalmusic.com

If you’re using a Wii on Windows, VJ Kung Fu has posted a full walkthrough:
Wii to MIDI Windows Walkthrough

And here’s an example on Vimeo of using the Wii with Processing: (I’m sure there are others)
Wii + Processing

Click to play

Mrmr is an open protocol for mobile devices. It is used to dynamically create user interfaces on your iPod Touch or iPhone which respond to client apps in a multi-user performance environment.

Okay, that sounds awfully dry. Let’s try that again.

Mrmr lets you control Quartz Composer applications (or really , any compatible OSC implementation)over Wi-Fi from your iPod Touch or iPhone. Now you, too, can dance around like a lunatic while still controlling your visuals from the dance floor. Did we mention it’s multi-user, as well?

Mrmr is the brainchild of Eric Redlinger, researcher-in-residence at Brooklyn Polytechnic University’s Integrated Digital Media Program. He has leveraged the iPhone’s OS X underbelly and 10.5’s new Quartz Composer features to allow this sort of functionality.

I had the lucky* chance to interview Eric and ask him a few questions about Mrmr and the iPhone. Apologies for the quality of the interview, it was very spur of the moment.

Mrmr is a work in progress, but I think the results so far speak for themselves.

*(ok, no so lucky, my desk is right next to his, but somehow I had not seen Mrmr in action until just recently…)

Most of the emphasis on working with multi-touch and alternative controllers has been on our sister site, Create Digital Music. But in a way, visuals are even more demanding of new hardware. After all, musicians have all kinds of hardware that work perfectly for performance (keyboards, knobs, drums, violins, sousaphones, kazoos, and whatnot). But new visual performance media demand something different if they’re to evolve.

Oh yeah, that, and most pro visual apps are kind of a b**** to use with a mouse and aren’t all that much better with a tablet. (Unless you’re somehow discovered the secret and find a Wacom as easy to use as a ballpoint. Please, tell me how.)

That makes this tidbit all the more interesting:

Jazzmutant is proud to have been selected by the Siggraph Emerging Technologies Committee in San Diego to demo a new prototype device for digital imaging involving multi-touch control. This solution will go beyond mere finger-drawing and clearly illustrate a new way to interact and improve productivity with drawing and video editing software. Furthermore, the solution presented will be the very first multi-touch enabled Tablet PC shown to the public.

JazzMutant news

What’s that now? Visual editing on a multi-touch surface? JazzMutant is best known for the creation of the Lemur multi-touch hardware. It wasn’t specifically intended for music, but that’s where it got most attention; you can, incidentally, route its native OSC control to Processing, Max/MSP/Jitter, Pd/GEM, Flash, and so on. But it was pricey (US$2500), and while you could design your own interfaces for it, it wasn’t quite the same as having a computer.

Now we get a one-two punch of tantalizing possibilities: a controller specific to visuals, whatever that may mean, and the possibility of using an actual computer with multi-touch input. I’d love to have that with some of what I’m building with Processing these days for performance. I’m a little more skeptical on the visual hardware side, only because so far that has tended to mean a selection of templates for Lemur-like hardware. But either way, this is promising — we’ll be watching the news out of SIGGRAPH very closely indeed.

Adrien Mondot has posted a preview of his latest creation for OS X - eMotion. eMotion is an physically modeled, OpenGL powered particle engine enabled app that is Quartz Composer friendly, has an incredible text engine, and has Wii remote, wacom tablet, OSC and sudden motion sensor inputs to drive parameters. Watch the video to see for yourself, the text effects are quite amazing.

Unfortunately no public beta is currently available. You can check out Adriens other Quartz Composer and OpenGL related projects (with downloads and source) on his blog. Be sure to check out the “Simple Time Remap” application, it seems like the smoothest video scratching tool Ive come across.

eMotion looks to be an incredibly powerful and smooth addition to any VJ setup, especially with its Quartz Composer integration and text-savvy nature. This is one to keep an eye on. We’ll try and get you more details on eMotion, but for now you can lust after its mysterious ways and watch that video.