Apple’s iPhone — and the significantly more affordable, doesn’t-have-to-be-a-phone iPod Touch — are essentially pocket-sized, intelligent multi-touch controllers. Hooking them up to visual software as controllers simply requires some app on the phone to transmit data, and some way of dealing with that data on the computer side. We’ve already seen this a bit on Create Digital Motion, and we’ve been covering some of the specifics of parsing data with Pd (Pure Data), the open-source, tri-platform patching software, on Create Digital Music this week.

Here’s the basic setup:

On Your iPod/iPhone

You have two options of software to use on your iThing. (You’ll need to “jailbreak” your device, as these are not — and may never be, for all I know — approved Apple apps.)

1. mrmr by Eric Redlinger of Brooklyn (top right):  open-source, editable control screens (requires Mac-only software to edit). See our interview with Eric, including some examples with Quartz Composer.

2. akaRemote.app by Masayuki Akamatsu of Japan: not open-source, not editable, but comes with a set of useful control templates, and you can transmit data to the app. See our look at a recent release. Upcoming Mac-only visualist app 3L has its own special akaRemote-based bridge called i3L, which also runs on iPhone/iPod Touch; see our look at i3L with artificial eyes.

On Your Computer

While the iPhone and iPod Touch have Apple logos on them, all of these apps send OpenSoundControl data. That means any OSC-compatible software will work, which is gradually including more visual software, as well as modular apps like Quartz Composer, Max/MSP/Jitter, Pd/GEM, and vvvv. (I love saying that last one … vvvvvvvvvvvvvvvvvvv. Okay, moving on.)

Of all of these, Pd is the one solution that’s free, open source, and runs on any platform. That means it’s also a viable candidate for translating incoming OSC data to more broadly-compatible MIDI. (i3L has you covered, as it uses a MIDI bridge.)

We have not one but two sets of tutorials / example patches for working with Pd on Create Digital Music, using a patch like Cesare’s, pictured above:

Control Music and Visuals with iPhone/iPod, Free Via Pd

Tutorial: More iPhone/iPod Touch Control With Open-Source Pure Data

So, Is It Worth It?

I usually don’t ask that question, preferring instead to report on what other folks are doing. But it is always worth asking yourself — and it is an entirely personal question. I’m not totally convinced in the case of these devices that I’d want to buy one solely for VJing, but then, what makes this so cool is that it adds on additional functionality to a device. (Too bad Apple is being so uptight about third-party development, but at least there’s an SDK — and plenty of hackers ready to break Apple’s rules.)

My own preference remains squarely with tangible controllers and tactile feedback, especially as some of the advantages of multi-touch are diminished by the iPod/iPhone’s diminutive size. But I absolutely see the argument for using these. What do you think, dear readers?

video mlr clone demo from themoves on Vimeo.

I’ve always been intrigued by the idea of using the Monome, the popular open-source, button-bestrewn controller device, for visuals and not just sound. But I haven’t seen any examples in person. Via comments on Monday’s Live + Isadora story, here’s one example by Joshua Nugent. The software is inspired by MLR, the sound sample-rocking patch that comes with Monome. In fact, that’s one area in which I think the Monome hasn’t gotten enough credit — a lot of its popularity actually comes from its software design, not just its elegantly-minimal hardware design.

I tend to do visuals differently, so I’ll have to finish my kit, finally, and post some software examples. And yes, while this is done in Isadora, there’s nothing saying you couldn’t use other software tools.

Bonus points for including Bush dancing in Africa. Now someone send poor Josh a screen and some windowblinds.

More on this project on the Monome forums, including instructions:

how about a video version of MLR/64step?

I’d love to see more Monome examples, so if you’ve got them, send them in!

As digital musicians have realized for some time, working with computers is all about physical control. It’s the difference between feeling like you’re operating software and playing an instrument. So it’s no accident that Jay Smith is quick to call the Ohm, a new hardware controller for visuals and music, an “instrument.” I got to hang out at the Hoboken, New Jersey office of Livid and play with the Ohm a bit. Hands-on experience is everything: as you can see, you’ve got a nicely-crafted wooden crossfader piece, for starters. Here are some first impressions.

More reasons you love OSC? You can get input from a SpaceNavigator 3D mouse, and translate the data into Max/MSP/Jitter, Quartz Composer, VDMX5 (as mentioned earlier today), Processing, and other tools — plus everything else, via MIDI. The solution: the inexpensive Mac utility OSCulator, covered this evening over on CDMusic:

OSCulator for Mac: Alternative Control, Now with 3D Mice, AppleScript, Combos

3D mice, eh? Suddenly navigating those 3D generative graphics built in Processing for a live performance starts to sound more appealing, huh?

Earlier today:

VDMX5, Now with OpenSoundControl - Everyone Else, You Listening?

So, yeah, we will have to do that “dummie’s guide” to OSC as promised earlier!

VDMX, the “realtime video studio” from the makers of Grid, semi-modular live visualist tool for Mac, and very possibly the “world’s most mature beta software”, continues to pack goodness into each new beta release. But the most recent addition, just now available as part of the primary public beta release, warms my heart. The latest beta adds support for OpenSoundControl or OSC, a protocol for communicating between computers, hardware, and software. With high-resolution data, an open-ended naming scheme, and network savvy, OSC is just plain better than MIDI at at least some tasks. But it’s suffered from a lack of documentation, a vague specification that’s a bit too open-ended and daunting to developers, and most of all, a lack of critical mass as far as tools. One small step at a time, that’s changing.

So far, the VDMX developers have tested their implementation with the multi-touch Lemur controller and inter-app data with Apple’s free Quartz Composer. But OSC is also supported in Processing, Max/MSP/Jitter, Pure Data, and free controller implementations for the Wii remote and Wacom tablet, among many other things. It’s also supported natively by the Monome controller, which is quite a lot cheaper than a Lemur. Want to pipe data from a Wacom into Quartz Composer? Sync data between a Mac running VDMX and a PC running vvvv? Something … uh … else that I haven’t thought of? Quite a lot is possible.

And an iPhone implementation could come next. (Or you can send OSC from iPhone right now, with MrMr.)

You can try this out in the beta that’s available now, though David Lublin from Vidvox warns us:

…there’s still more to do with OSC, currently no way to trigger files, it’s just control data, but the core functionality for sending and receiving is there, so we really just need to start building on top of that for whatever protocols there are to support

…the OSC support is still very much in it’s early stages.. it is there and stable, but we are barely scratching the surface of what we can do with it

Of course, that description could apply to OSC in general, which is why the two CDM’s (createdigitalmusic and createdigitalmotion) will be working over the course of 2008 to work with users and developers to share knowledge and get better implementations in everyone’s hands sooner. It’s something I hope to check in on while I’m at NAMM next week. Stay tuned.

Vidvox + VDMX5

VDMX5 public beta direct download link (Mac only)

Introduction to OpenSoundControl [opensoundcontrol.org]

Speaking of multi-touch OSC implementations, we hope to catch up with CyberPatrolUnit, who’s working on a VJ interface on Lemur, as pictured below. He’ll be bringing it to the CDM NAMM party, so if you’re in LA, come see us next Friday!

Musicians have thousands of years of history when it comes to interfaces and instruments, but visuals are relatively new. Little wonder, then, that visualists are eager to try new interfaces to help make visuals akin to performance instruments. Or, in less lofty terms, let’s get Wii remote wagging in the club tonight.

Over on createdigitalmusic.com, we’re celebrating Game Day — basically, I’m squeezing as many game-related posts into 24 hours, because a whole bunch of tips came in at once. A couple of Wii-related controller solutions jumped out.

Wii + Flash

Getting hardware control in Adobe Flash requires some work: basically, you need a client-server model. The good news is, there are already two Wii-specific solutions out there.

MoteDaemon, Mac OS X
WiiFlash.org, Windows Blog, Download, Google Code

Pretty soon, people are cooking up Minority Report demos with Flash (using Papervision for 2.5D-style 3D in Flash’s 2D world, and Open Dynamics Engine for physics):

I’d loved to see this coupled with something like Onyx for an all-Wii, all-Flash performance app. With Flex and AIR (Adobe Integrated Runtime), this could be the basis for some really hard-core, full-blown apps … though you will be limited by Flash’s slower performance, at least in comparison to C/C++-based tools like Max or even Java.

Looks like Linux users are presently out of luck, unless I’m missing something (feel free to chime in if you know a cross-platform alternative).

Wii + MIDI, OpenSoundControl (OSC)

For a more app-agnostic solution, you can hook up a Wii to send MIDI or, ideally, OpenSoundControl (OSC) to apps that support it (vvvv, Max, Pd, and hopefully VJ apps soon — I’ve heard a couple of developers working on it).

On Mac, you can send both OSC and MIDI with one app, perfectly-suited to taking data from the Wii controller:
OSCulator. OSCulator is a great tool for the Wii, but it also shows promise of what a hardware input hub could generally look like, with open-ended inputs controlling visuals and sound rather than pre-defined, MIDI-style keyboards and knob and faderboxes as have traditionally been used in music.

On PC, GlovePIE is an awesomely-powerful scripting tool for use with Wii and other game devices. I’ve talked about it endlessly before, but I’ll stay quiet — just go. Get it. Enjoy. In fact, with OSCulator on Mac and GlovePIE on Windows, it’s hard not to be insanely happy on either platform.

Another interesting out-of-the-box alternative, though, is the new Wiinstrument. It’s largely geared for playing back drum samples, but it’s not hard to take that metaphor and use those control changes and other MIDI messages as visual controls — especially if you think of your “drum kit” as sets of visual clips (video or other visuals).

Wiinstrument on createdigitalmusic.com

If you’re using a Wii on Windows, VJ Kung Fu has posted a full walkthrough:
Wii to MIDI Windows Walkthrough

And here’s an example on Vimeo of using the Wii with Processing: (I’m sure there are others)
Wii + Processing

Mobile, touch control of visual apps? We’ve seen one way to do it; here’s an even more compatible MIDI approach. I think I’d stick to the iPod Touch, personally, just to conserve battery on a phone. But it’s interesting. -PK

Hot on the heels of Mrmr, ArtificialEyes (the Istanbul-based VJ collective) have released i3L MIDI Bridge for the iPhone, which builds on Masayuki Akamatsu’s aka.remote.app, giving those without Max/MSP skills a simple way to bridge their iPhone and audio/video performance software of choice.

i3L (pronounced “i thrill”) is Freeware, and was developed using Max/MSP from Cycling74 and is a support application for aka.remote by Masayuki Akamatsu i3L receives pre-defined UDP messages from aka.remote.app running on the iPhone, scales the values to MIDI, and allows you to configure the sending MIDI channel and control change message number. While this software was developed to work seamlessly with our Real-time 3D VJ software Thrill, you can use this program with any audio or video software which receives midi messages.

If you give i3L a try, let us know how it works out for you in the comments.

artificialeyes.tv
aka.objects by Masayuki Akamatsu — must-have stuff for Max/MSP/Jitter!

Ed.: Does anyone know if multi-touch gestures would be possible in DIY apps? Maybe in February when Apple shares the official SDK? -PK

Click to play

Mrmr is an open protocol for mobile devices. It is used to dynamically create user interfaces on your iPod Touch or iPhone which respond to client apps in a multi-user performance environment.

Okay, that sounds awfully dry. Let’s try that again.

Mrmr lets you control Quartz Composer applications (or really , any compatible OSC implementation)over Wi-Fi from your iPod Touch or iPhone. Now you, too, can dance around like a lunatic while still controlling your visuals from the dance floor. Did we mention it’s multi-user, as well?

Mrmr is the brainchild of Eric Redlinger, researcher-in-residence at Brooklyn Polytechnic University’s Integrated Digital Media Program. He has leveraged the iPhone’s OS X underbelly and 10.5’s new Quartz Composer features to allow this sort of functionality.

I had the lucky* chance to interview Eric and ask him a few questions about Mrmr and the iPhone. Apologies for the quality of the interview, it was very spur of the moment.

Mrmr is a work in progress, but I think the results so far speak for themselves.

*(ok, no so lucky, my desk is right next to his, but somehow I had not seen Mrmr in action until just recently…)

Pioneer have broken new ground in overstatement with the announcement of their upcoming Video/Audio mixer, the US$6000 SVM-1000.

World’s first mixer for 4-channel audio and video.

I guess they completely missed the AVM02 then? I’m not sure how, it’s rather big and heavy.

Does anybody actually believe marketing people? Do they even believe themselves when they open their mouths? A conversation between two marketing people must be like facing two TVs towards each other, tuning one to MTV and the other to MTV Europe. But I digress. Specifications can’t lie (much), so: