Apple’s iPhone — and the significantly more affordable, doesn’t-have-to-be-a-phone iPod Touch — are essentially pocket-sized, intelligent multi-touch controllers. Hooking them up to visual software as controllers simply requires some app on the phone to transmit data, and some way of dealing with that data on the computer side. We’ve already seen this a bit on Create Digital Motion, and we’ve been covering some of the specifics of parsing data with Pd (Pure Data), the open-source, tri-platform patching software, on Create Digital Music this week.

Here’s the basic setup:

On Your iPod/iPhone

You have two options of software to use on your iThing. (You’ll need to “jailbreak” your device, as these are not — and may never be, for all I know — approved Apple apps.)

1. mrmr by Eric Redlinger of Brooklyn (top right):  open-source, editable control screens (requires Mac-only software to edit). See our interview with Eric, including some examples with Quartz Composer.

2. akaRemote.app by Masayuki Akamatsu of Japan: not open-source, not editable, but comes with a set of useful control templates, and you can transmit data to the app. See our look at a recent release. Upcoming Mac-only visualist app 3L has its own special akaRemote-based bridge called i3L, which also runs on iPhone/iPod Touch; see our look at i3L with artificial eyes.

On Your Computer

While the iPhone and iPod Touch have Apple logos on them, all of these apps send OpenSoundControl data. That means any OSC-compatible software will work, which is gradually including more visual software, as well as modular apps like Quartz Composer, Max/MSP/Jitter, Pd/GEM, and vvvv. (I love saying that last one … vvvvvvvvvvvvvvvvvvv. Okay, moving on.)

Of all of these, Pd is the one solution that’s free, open source, and runs on any platform. That means it’s also a viable candidate for translating incoming OSC data to more broadly-compatible MIDI. (i3L has you covered, as it uses a MIDI bridge.)

We have not one but two sets of tutorials / example patches for working with Pd on Create Digital Music, using a patch like Cesare’s, pictured above:

Control Music and Visuals with iPhone/iPod, Free Via Pd

Tutorial: More iPhone/iPod Touch Control With Open-Source Pure Data

So, Is It Worth It?

I usually don’t ask that question, preferring instead to report on what other folks are doing. But it is always worth asking yourself — and it is an entirely personal question. I’m not totally convinced in the case of these devices that I’d want to buy one solely for VJing, but then, what makes this so cool is that it adds on additional functionality to a device. (Too bad Apple is being so uptight about third-party development, but at least there’s an SDK — and plenty of hackers ready to break Apple’s rules.)

My own preference remains squarely with tangible controllers and tactile feedback, especially as some of the advantages of multi-touch are diminished by the iPod/iPhone’s diminutive size. But I absolutely see the argument for using these. What do you think, dear readers?

iPhone development continues despite a lack of SDK. Masayuki Akamatsu has updated his native iPhone OSC client aka.iphone to 2.1, and includes plethora of screen/input configurations to suit your mobile performance needs. The update should suit typical VJ and Dj applications with its range of GUI sliders, knobs, button grids, etc.

If you have an iPhone or iPod Touch, be sure and check out aka.iphone. Very cool stuff. Yes, sorry, another Apple oriented update. Shoot me.

[Ed.: Note you will want Max/MSP/Jitter to work with this app, though I think it should work with Windows, too; I just lack a machine that can test it. Be sure to see also Masayuki's complete Max collection, which now covers everything from Wii remotes to 3D SpaceNavigator controllers, Apple Remotes, and even the motion sensor in the MacBook Pro. Completely brilliant. And needless to say, even if you don't want to hop on AT&T's mobile network, this gives you options with iPod Touch, too, even if you can't play Phase on that. -PK]

Mobile, touch control of visual apps? We’ve seen one way to do it; here’s an even more compatible MIDI approach. I think I’d stick to the iPod Touch, personally, just to conserve battery on a phone. But it’s interesting. -PK

Hot on the heels of Mrmr, ArtificialEyes (the Istanbul-based VJ collective) have released i3L MIDI Bridge for the iPhone, which builds on Masayuki Akamatsu’s aka.remote.app, giving those without Max/MSP skills a simple way to bridge their iPhone and audio/video performance software of choice.

i3L (pronounced “i thrill”) is Freeware, and was developed using Max/MSP from Cycling74 and is a support application for aka.remote by Masayuki Akamatsu i3L receives pre-defined UDP messages from aka.remote.app running on the iPhone, scales the values to MIDI, and allows you to configure the sending MIDI channel and control change message number. While this software was developed to work seamlessly with our Real-time 3D VJ software Thrill, you can use this program with any audio or video software which receives midi messages.

If you give i3L a try, let us know how it works out for you in the comments.

artificialeyes.tv
aka.objects by Masayuki Akamatsu — must-have stuff for Max/MSP/Jitter!

Ed.: Does anyone know if multi-touch gestures would be possible in DIY apps? Maybe in February when Apple shares the official SDK? -PK

Click to play

Mrmr is an open protocol for mobile devices. It is used to dynamically create user interfaces on your iPod Touch or iPhone which respond to client apps in a multi-user performance environment.

Okay, that sounds awfully dry. Let’s try that again.

Mrmr lets you control Quartz Composer applications (or really , any compatible OSC implementation)over Wi-Fi from your iPod Touch or iPhone. Now you, too, can dance around like a lunatic while still controlling your visuals from the dance floor. Did we mention it’s multi-user, as well?

Mrmr is the brainchild of Eric Redlinger, researcher-in-residence at Brooklyn Polytechnic University’s Integrated Digital Media Program. He has leveraged the iPhone’s OS X underbelly and 10.5’s new Quartz Composer features to allow this sort of functionality.

I had the lucky* chance to interview Eric and ask him a few questions about Mrmr and the iPhone. Apologies for the quality of the interview, it was very spur of the moment.

Mrmr is a work in progress, but I think the results so far speak for themselves.

*(ok, no so lucky, my desk is right next to his, but somehow I had not seen Mrmr in action until just recently…)

Nice work if you can get it: visualist Toby / tobyz, aka *spark, had one heck of a live visual rig for the Geneva Motor Show’s Ford booth. One ongoing challenge for live VJs is making it clear what they’re doing. Solution: “VJ Crew” t-shirts, lovely women (can’t hurt), and, of course, giant touchscreens for interfacing with the Mac software. Live visuals made the client happy (interactive text!) and made guests happy by snapping photos (happier clients!). The software rig was the glue:

• VDMX5 from Vidvox (still in beta), which looks the ideal interface for the work they were doing. Custom, modular palettes and windows also lend themselves nicely to the touchscreen.
• Quartz Composer patches, built with Apple’s free visual-patching developer tool for custom visuals, integrate directly into VDMX5. Result: a modular, custom system that works live and gets the job done!

Documentation:
Photos and notes on the install/performance
Video montage of the event

Open Source Tool for Easy, Live Titling

But you don’t have to just silently drool over this setup. Toby is nice enough to give away one of the tools he built to make it happen, open sourced so the community can improve upon it.

Spark Titler, GPL-ed Quartz Composer patch for live text titles

Quick titling? No problem. Toby describes it thusly:

the titler’s interface allows you to take between two sets of title/subtitle, with the choice of four backgrounds: black / green / a quicktime movie or a folder of images. the output window will automatically go full-screen on the second monitor if it detects one is available at launch, otherwise it will remain a resizable conventional window.

it is released with the intention that it can be reused for other events without changing a single line of code: you can design the animation and incorporate quicktime movies in the design by editing the ‘GFX’ macro in the quartz composer patch, and its a matter of drag and drop replace the logo in the interface.

And if you’re up for patching some improvements (with a little light Xcode use), you can dig into the source, as well.

Why Live and Interactive Rules

But enough about the technology. Part of the whole philosophy of this site is that we believe “rendered”, wonderful as it can be, sometimes must make room for “live.” Quartz Composer is just one of a generation of tools that allow visualists to move in this direction. It’s part of our interest not just in the “Final Cut” (ahem), but the live cut. Artists are moving in this direction for creative reasons, but it’s equally encouraging to see it working well on gigs. It’s a practical, technical issue as much as it is a philosophical one. Toby explains in a caption to a recent blog entry on his Quartz Composer titling patch (pictured here):

this picture is but a snapshot of the revolution. it really feels like that. a real let-down of the geneva motor show pre-production was the inability to translate the creative agency’s after-effects rendered text animations into the live, dynamic setup. there just was no way to implement anything vaguely sophisticated without seeing the framerate drop to near zero. structure record, something driven by video sampling and seemingly tangental [sic] to text rendering, is the key to solving that problem… and so here it is solved, as if on cue for the frankfurt motor show.

(emphasis mine)

Structure Record looks like some kind of custom patch. Not sure how it works — Toby, if you want to enlighten us, please do!

So have a look at that patch, and viva la revolución!

(Via the Quartz Composer dev list and vade.)

Most of the emphasis on working with multi-touch and alternative controllers has been on our sister site, Create Digital Music. But in a way, visuals are even more demanding of new hardware. After all, musicians have all kinds of hardware that work perfectly for performance (keyboards, knobs, drums, violins, sousaphones, kazoos, and whatnot). But new visual performance media demand something different if they’re to evolve.

Oh yeah, that, and most pro visual apps are kind of a b**** to use with a mouse and aren’t all that much better with a tablet. (Unless you’re somehow discovered the secret and find a Wacom as easy to use as a ballpoint. Please, tell me how.)

That makes this tidbit all the more interesting:

Jazzmutant is proud to have been selected by the Siggraph Emerging Technologies Committee in San Diego to demo a new prototype device for digital imaging involving multi-touch control. This solution will go beyond mere finger-drawing and clearly illustrate a new way to interact and improve productivity with drawing and video editing software. Furthermore, the solution presented will be the very first multi-touch enabled Tablet PC shown to the public.

JazzMutant news

What’s that now? Visual editing on a multi-touch surface? JazzMutant is best known for the creation of the Lemur multi-touch hardware. It wasn’t specifically intended for music, but that’s where it got most attention; you can, incidentally, route its native OSC control to Processing, Max/MSP/Jitter, Pd/GEM, Flash, and so on. But it was pricey (US$2500), and while you could design your own interfaces for it, it wasn’t quite the same as having a computer.

Now we get a one-two punch of tantalizing possibilities: a controller specific to visuals, whatever that may mean, and the possibility of using an actual computer with multi-touch input. I’d love to have that with some of what I’m building with Processing these days for performance. I’m a little more skeptical on the visual hardware side, only because so far that has tended to mean a selection of templates for Lemur-like hardware. But either way, this is promising — we’ll be watching the news out of SIGGRAPH very closely indeed.

Yes. You can now VJ from your iPhone, controlling your Max/MSP/Jitter patch via Safari and the new aka.iphone object from Masayuki Akamatsu of aka.wiiremote fame.

The aka.iphone object is a work in progress, but stay tuned, there’s much fun to be had. Can anyone say “pocket Lemur”?

The blogosphere this week is all abuzz about the supposedly desire-inducing Upravlator. The awkwardly-named hardware comes from Art Lebedev, the mysterious designer who first promised the Optimus Keyboard, a unique “design concept” with tiny color displays under each key. That indeed sounds cool, but instead, after months of delays and promises, the shipping product turned out to be the Optimus Three, with three little displays that double as buttons. Full keyboard with displays: interesting. Three display buttons with no real function: erm? Instead of spending about US$150 on an Optimus Three, why not a Nintendo DS Lite? Which do you think would be more useful?

This week, we get the Upravlator. Imagine a powerful interactive piece of hardware that connects to your computer’s video port and dynamically displays, in full color, everything from interface widgets to graphics and video to dynamic Web content, all completely customizable for your own needs. Sounds unbelievably useful, right? Good thing we already have such a device: it’s called a monitor. Want touch input? It’s called a touchscreen monitor. Now take that monitor, divide it up into a tiny 4×3 grid, eliminating a significant portion of its usable space. Replace the full resolution of the touchscreen with 12 buttons (thousands of levels of resolution reduced to a dozen). Put it in a big bulky case, wait until 2007 to ship it (presumably for some enormous price), and require developers to rewrite software to use it. Now you’re getting the picture:

Upravlator Product Announcement [artlebedev.com official site]
The Upravlator unveiled [DVguru]
Art Lebedev explains Upravlator to five year olds, no one else [Engadget]
(and, like a zillion other blogs)

Worst of all, the Upravlator takes up lots of desk real estate and a free VGA port — two things better dedicated to a real monitor, especially when touchscreen monitors are coming down in price.

The dynamic keyboard remains a cool product. It may come in at an astronomical price, but the concept is good: take the physical feedback and ease of a great hardware input device (keyboard) and add dynamic visuals to make it more flexible. By contrast, the Upravlator and the Optimum Mini Three are useful neither as displays nor as input devices and actually reduce efficiency. We’ll wait for the Optimus-113 keyboard, if it can actually ship.

Don’t be too sad, though. You want gadget lust? Chumby’s got your gadget lust. You’ll be hearing a lot more about this homebrewed, open source, hackable gadget soon, because Team CDMo desperately wants one right now. Let’s compare:

As great as the potential of advanced touchscreens may be, for music and other media applications, touchscreens aren’t much fun to touch. Close your eyes and remove visual feedback, and you’re basically running your finger along a piece of plastic. (You’d think we could figure out a way to at least texture it without losing tracking.) Compare that to piano or drums: musical instruments can be played satisfyingly with your eyes closed. Yeah, you can do that to look “deep,” but the point is, you’re relying on tactile, not visual feedback.

Here’s a promising solution: the Hyperfabric project (via the fascinating ramblings at SteamSHIFT). This stuff is strong (it can support body weight), and lets you actually touch, squeeze, grab, and otherwise manipulate a large-scale fabric surface to control computer-generated imagery. It’s certainly workable as a musical instrument, if you want to be able to, in their words, “press your face into the hyperfabric to release fairies.”

I have no idea how this thing works, though I’m guessing some kind of correlation of pressure with video sensing. It’s commercially available, or you can just ponder what giant spiderweb-like surfaces might someday do for music.

Via Pixelsumo comes a fantastic lineup of links to futuristic interfaces for music, etc. If you were impressed by Cycling 74’s new Lemur touchscreen, with the power to let you touch multiple points on a pad simultaneously for controlling music/sound/etc., Jeff Han has built his own. His Frustrated Total Internal Reflection project may sound like some sort of existential experience, but it’s really an optical system for tracking multiple finger touches on a screen. You need a back projector, so it’s not as portable or compact as the Lemur, but watch the video: extremely low latency, extremely high sensitivity. Jeff has other tricks up his sleeve, too: using LEDs for touch control, and volumetric 3D displays using dust, among others. I’m heading to NYU later this week, so I’ll have to see if there’s something in the water. If you’re at Siggraph, go check these out and let us know more!