MSAFluid for processing (Controlled by iPhone) from Memo Akten on Vimeo.

We can continue to ponder how to convince Apple to let Memo’s simple but powerful-looking MSA Remote multitouch app on the App Store. But in the meantime, a resourceful developer has tried simply writing a quick app for the Safari browser. This is doubly promising to me. I love full-blown apps, and they typically allow access to some of the powerful sensor and location features of mobile devices. But that’s not to say browser apps won’t also have a place for quick prototyping, live performance, and installation. WebKit browsers are now not only on iPhone and iPod touch, but Palm Pre and Android – and, I suspect, more places soon. This could be a great outlet even for extending functionality of apps.

Anyway, if you’re looking for a quick way of using the TUIO protocol – as Memo is doing with his (App Store-rejected) app + Processing above – Andrew Turley’s app is a quick fix. I’ll be looking at mobile browser development alongside app development, I know, and I imagine all of us will keep praying for the MSA app.

TUIO Multitouch for iPhone: Browser App Hack Replaces Rejected App [Create Digital Music]

touchy feely [Pillowsopher Blog]

And yes, as this is a browser app, it should work on other platforms, too. The disadvantage of Android G1 is you’ll get only one-touch … while we wait for generally-available multitouch capabilities on Android, I imagine more specialized apps with specific platform tie-ins will be more useful.

  • akuAtaja

    coming soon for android I hope

  • Peter Kirn

    Well, not much point with Android just yet — not until it supports multitouch. As far as I know, the browser app will work on Android right now, but there's not a whole lot of use to employing TUIO if you don't have multitouch.

    Once there's multitouch on Android, you can bet we'll have a BUNCH of things to do with it. (heck, I'll write something if no one else does!)

  • Andrew Turley

    @akuAtaja, I don't have an Android phone to test with, so I'm not sure whether or not the browser supports the ontouchstart, ontouchchange and ontouchend message listeners in JavaScript. If not, one could add onmouseup and onmousedown handlers to the touch objects, and have those handlers put appropriate messages on the message queue.

    There may not be as much that you can do on Android without multitouch, but I still think people might enjoy at least being able to use the phone screen to control apps on a host computer.

  • Peter Kirn

    Right, and technically there's only *one* thing you can't do on the Android right now – manipulate two controls at once. On the other hand, I've seen so many faux fader apps at this point, I rather wonder what could be done with designing other elements for interaction that use single touch. Given a virtual fader never feels as good as a real one, I think we have a lot more to explore.

    Andrew, do you know where those specs come from in JavaScript? Are they an Apple extension, or is there some sort of standard for event listeners? (I know it's very, very possible to extend JavaScript … curious if there is a standard for this. I just don't know JS all that well.)

    It'd be really great, for instance, if there were standard tilt / accelerometer / location listeners *in the JavaScript* implementation. Most of these devices are pretty similar – at the hardware level, I mean, the Android isn't all that different from the iPhone, ditto other similar devices.

  • Andrew Turley

    @Peter, I totally agree about the virtual faders. Part of the reason I went down this path was because I wanted to try out some different interface designs, and I didn't want to have to jump through all the hoops required to get an app onto the phone. Rapid deployment is part of a rapid development cycle.

    As for the ontouch* and ongesture* events, I'm pretty sure that these extensions are Apple-specific, but other vendors could implement them if they chose to do so. Standardizing these events (and the tilt/accelerometer/location things you mentioned) would be great, since, as you point out, most of these devices are the same on some level. It may just take a little while. Look at how long it took things like canvas and xmlHttpRequest to become widely supported.

  • Peter Kirn

    Oh, and that's nothing against virtual faders — I've used them myself onstage! ;) I just could see some interesting stuff happening across the range of things you could do, and perhaps *not* having multitouch on Android – which I imagine is likely only temporary – could force us to try some other stuff.

    And yeah, these kinds of event listeners would be useful even on, say, a netbook, etc.

  • Ryan Nestor

    If you want IPhone .app TUIO right now… Why not try OSCemote from the AppStore?

  • Peter Kirn

    @Ryan: because that uses OSC, not TUIO per se? ;) Different protocols. But yes, OSC is a great way to go. That just also begs the question of why Apple rejected a TUIO app but not OSC apps (not that I want to give them any more ideas…)

  • Andrew Turley

    @Peter, OSCemote actually does output TUIO messages when you use their multitouch setting.

    @Ryan, if somebody wanted to use TUIO then I think that in many cases OSCemote would be a much better solution than the one I presented. Making something that worked with TUIO gave me a target to shoot for. That being said, my system does have the advantages of (1) being free and (2) not requiring a user (the person interacting with the TUIO-enabled program) to download new software onto their phone. This could be useful in a situation where you are showing off a TUIO-enabled program and you want strangers to be able to interact with it. I think it's a lot easier to say, "Connect to this network and go to this URL" than, "Go to the app store, download this app that costs $4.99, open the app, navigate to the setting screen, type in this port and this address, then go to the multitouch screen."

    Which system you use depends on your needs. I think that's why, from a user's point of view, the iPhone App store is somewhat problematic, because Apple is in a position to dictate which solutions are appropriate for everyone. So even if OSCemote and MSA Remote both send TUIO messages, one may be more appropriate than the other for a given use. And having choices lets you do things that nobody else thought of.

    And just to be clear, OSCemote is a great app and totally worth the price. If you are serious about working with OSC on the iPhone, you should have this app.

  • Pingback: TUIO Multitouch Control on the iPhone: Now Via a Browser Hack, Since the App Was Rejected «

  • Ryan Nestor

    @A.T. Thanks for clarifying. PLUG: I use mrmr too, for OSC connectivity, but they do not yet support TUIO. Mrmr is free and began its life at IDMI.

  • Peter Kirn

    Whoops, sorry. So, yes, I guess OSCemote does do that — my bad.

    And OSCemote is indeed awesome. I've got basically all the OSC apps on my iPod touch now. mrmr has some interesting features, as well, in particular. We've actually just set up the mrmr folks on our new site noisepages:

    – the goal there being, long run, not only supporting mrmr better but all OSC / TUIO / multitouch development.

    I suppose you could also just transmit OSC and translate to TUIO on the client side.

    @Andrew, I will say, my understanding is that Apple is not in any way curating some apps as better than others. I think they just didn't properly get this app. And I'm guessing, based on the feedback I've seen, that the review process just isn't completely consistent at this point.

    Of course, speaking of tradeoffs, this is part of why the iPhone is on AT&T and Google Android is not, because whatever the review process is, some carriers prefer that level of control.

    I will say, I think the ideal is basically what we have with Android: a specific app store for ease but without that app store being exclusive, the ability of the user to load apps themselves if they like, and the freedom to replace even default phone functions if you think you can do better. That's not to say I think Android is better than iPhone – just that it's my own personal preference on one axis. (On the contrary, I think iPhone/iPod touch bests Android the platform and G1 the currently-available handset in other ways.)

    But, anyway… bottom line is, in the long run we ought to have both native and browser-based apps for a lot of mobile platforms, and that's what's most important. Also, I remain hopeful that Memo's app will get onto the store, too, because choice is good.

  • Peter Kirn

    Side note: yes, Google has removed some apps from their store, too, based on carrier concerns. The key difference is that Google allows you to install apps from other sources if you like.

    You can flip this around, though, and say – hey, wouldn't it be great if we had a *desktop* app store for our existing OSes? Imagine if you could run to that app store and buy visualist artworks. This has worked wonders for, say, games from Valve/Steam. You get unique multiuser capabilities, and almost none of those games is exclusive.

  • Pingback: TUIO multitouch on iPhone via browser hack - Hack a Day

  • Pingback: ionoi » Multitouch Control on the iPhone

  • Pingback: links for 2009-04-23 « Blarney Fellow

  • Pingback: Jacob Berendes » MAS - EI - Week 4

  • akuAtaja

    I found an app for android, which allows you to control your laptop via wifi. It is called gmote ( That means I am able to control ("mono-touch" only of course) msafluid in my laptop browser, with my g1. at least something :)