image Apple’s iPhone — and the significantly more affordable, doesn’t-have-to-be-a-phone iPod Touch — are essentially pocket-sized, intelligent multi-touch controllers. Hooking them up to visual software as controllers simply requires some app on the phone to transmit data, and some way of dealing with that data on the computer side. We’ve already seen this a bit on Create Digital Motion, and we’ve been covering some of the specifics of parsing data with Pd (Pure Data), the open-source, tri-platform patching software, on Create Digital Music this week.

Here’s the basic setup:

On Your iPod/iPhone

You have two options of software to use on your iThing. (You’ll need to “jailbreak” your device, as these are not — and may never be, for all I know — approved Apple apps.)

1. mrmr by Eric Redlinger of Brooklyn (top right):  open-source, editable control screens (requires Mac-only software to edit). See our interview with Eric, including some examples with Quartz Composer.

2. by Masayuki Akamatsu of Japan: not open-source, not editable, but comes with a set of useful control templates, and you can transmit data to the app. See our look at a recent release. Upcoming Mac-only visualist app 3L has its own special akaRemote-based bridge called i3L, which also runs on iPhone/iPod Touch; see our look at i3L with artificial eyes.

On Your Computer

While the iPhone and iPod Touch have Apple logos on them, all of these apps send OpenSoundControl data. That means any OSC-compatible software will work, which is gradually including more visual software, as well as modular apps like Quartz Composer, Max/MSP/Jitter, Pd/GEM, and vvvv. (I love saying that last one … vvvvvvvvvvvvvvvvvvv. Okay, moving on.)

Of all of these, Pd is the one solution that’s free, open source, and runs on any platform. That means it’s also a viable candidate for translating incoming OSC data to more broadly-compatible MIDI. (i3L has you covered, as it uses a MIDI bridge.)


We have not one but two sets of tutorials / example patches for working with Pd on Create Digital Music, using a patch like Cesare’s, pictured above:

Control Music and Visuals with iPhone/iPod, Free Via Pd

Tutorial: More iPhone/iPod Touch Control With Open-Source Pure Data

So, Is It Worth It?

I usually don’t ask that question, preferring instead to report on what other folks are doing. But it is always worth asking yourself — and it is an entirely personal question. I’m not totally convinced in the case of these devices that I’d want to buy one solely for VJing, but then, what makes this so cool is that it adds on additional functionality to a device. (Too bad Apple is being so uptight about third-party development, but at least there’s an SDK — and plenty of hackers ready to break Apple’s rules.)

My own preference remains squarely with tangible controllers and tactile feedback, especially as some of the advantages of multi-touch are diminished by the iPod/iPhone’s diminutive size. But I absolutely see the argument for using these. What do you think, dear readers?

  • ilan

    It's taken how long for the keytar to finally not be a joke when someone uses it on stage?

    Ok. I am being harsh. One of the first things that excited me about the iPhone was the prospect of being able to do visuals with it. Actually, less the iPhone and more the iPod Touch that interests me more.

    People are already using the iPhone to control modul8. I actually thought I saw this here… guess not

    I like the shaking bit :)

    All that typing and I don't plan to buy one in any near future.

  • Sam

    theres a lot of scope for this device- just look at some of the new games being made with the SDK. then there's also the whole accelerometer thing…

    if you can control more than one slider at once, then its better than a mouse, regardless of screen size imo(for visuals anyway). there's not much tactile feedback from a mouse anyway.

    the 2g iPhone is rumoured to be out in june/july, and its rumoured to have small vibrations on touch for feedback. if thats true, the mouse will go away and the iPhone will play. (if I ever get one, that is)

  • Pingback: Create Digital Music » iPod Touch/iPhone for Music Round-up

  • visceralX stuff, but…why can't i find stuff like this for my touch screen Pocket PC? how come this comes out for an iPhone with locked up 3rd party support, but nothing for the PPC, where you can develop till the cows come home. k, its not MULTI-touch, but still…you'd think that would make it even easier.

    well i guess i better shut up and just do it myself eh? = )

  • ilan

    Hear hear for tactility! I have access to a Lemur. Pricey piece of hardware and I would more likely go for a monome instead. Forget about the fact that I can barely get the Lemur to connect to my computer. Arrgh.

  • netchaiev

    I have been playing with those softs for a while.
    I haven't found the time to fully program i3L w/ akaremote though but Mrmr is great in parties and stages!
    Wii/iPhone control on stage of multimedia perf is awesome.

  • Pingback: Create Digital Music » iPhone/Touch Roundup: BtBx Acid Bass, iDrum Workflow and Babies, OpenSoundControl App