Kamel Makhloufi posts this appetite-whetting image of the “augmented reality” ARToolKit working on the iPhone platform. Sure, Microsoft and Sony may be hard at work on computer vision applications that don’t require tags, but for mobile applications, tags seem perfect. This is just a preview, but it already suggests some interesting art applications, as well as the usual games and such. I’ve been impressed with how well the camera on my Android-powered TMobile G1 works, even in low light, so the fact that tags are designed for easy readability by digital cameras suggests this could be eminently practical.
I just hope we also see ARToolKit on Android, too, naturally.
That was fast – Kyle McDonald notes in comments that some folks have already baked some parallel tracking and mapping system work on the iPhone, for markerless tracking. This sort of confirms for me why I like markers – there’s just so much data in this image to deal with, and presumably you have more you want to do in an app than just take care of tracking.
Having flowers pop out of this does look beautiful, and there is something magical about watching this happen without the markers.
By the way, since we’re looking at yet another iPhone video, I’m not advocating working on Android out of some sort of sense of ethical purity. Put simply, I just personally see the long-range development experience and deployment as being more flexible and broader. And I don’t think that thinking “mobile” has to be about any one platform. Ideally, you’d come up with solutions that allows you to get your artwork or game or whatever it is on iPhones and Androids and other devices.
reacTIVision is able to work thanks to these alien-looking markers called “fluidicials.” Karsten Schmidt aka toxi has developed software for creating more of these markers, and it generates characters like this “teddy bear” seen here.
Computer vision for tracking movement is cool. But add the ability to track actual objects, and you can extend the possibilities for interfaces. We’ll be playing around with this on our upcoming tangible hackday. A lot of the reason these experiments can proliferate is the availability of free frameworks that make the technology accessible to artists and designers. The tricky tracking work is done, leaving you to focus on where this tracking might actually be useful.
The other good news: while doing projected visual feedback or fancier tracking can get more complex and costly, if you just want to track some objects, all you need is a USB or FireWire camera and some printed stickers. Cost: $40 with a webcam, about $5 without.
Recently, one of the most popular of these libraries got a big update: reacTIVision 1.4. It’s the open-source, multi-platform framework that powers the reacTable, and was developed by Martin Kaltenbrunner and Ross Bencina at the Music Technology Group at the Universitat Pompeu Fabra in Barcelona, Spain.
Multi-platform really means multi-platform. It works with FireWire and USB cameras, Mac, Windows, and Linux, and has clients for C++, Java, C#, Processing, Pure Data, Max/MSP/Jitter, Quartz Composer, and Flash, plus a wide range of applications that support the OpenSoundControl-based TUIO protocol.
I’ll actually be testing both the tracker in reacTIVision and the Trackmate tracker from the LusidOSC project. The Trackmate software is built in OpenFrameworks. It does use a different protocol (LusidOSC), but that’s also based on OSC, and there’s even a tool that translates to TUIO.
For an example of what this all looks like when assembled - and some of the power of having a framework on which to build - here’s a tangible interface for a multiplayer game. It’s Pong with objects.
This games uses reactivision software, along with Flash, to detect symbol fiducial block movements. The game is played by moving these symbols on a table. Players can enter and exit the playing field at any time. The game adapts to the number of players. The lower the score the better; the first player with a score of 12 ends the game.
Music: Waterdrops by Yohan Shin http://www.geocities.com/cerup2
There’s more progress coming in reacTIVision world, too. First up: reacTIVision 1.5. Martin tells us:
After this release I am now implementing reacTIVision 1.5, which will improve the multi-touch tracking performance, and already implement the upcoming TUIO 1.1 blob tracking extensions, for the transmission of basic untagged object descriptors.
Following that, the next plan is for TUIO2, an expanded protocol that will address some of the shortcomings of the first version, to be released with a future update to reacTIVision. You can read the full specification for the new protocol, but Martin has kindly given us a Cliff Notes version:
To summarize, TUIO2 has a flat profile, which now includes symbol, cursor and blob descriptors in much more detail. Symbols now also can carry content info (e.g. datamatrix), cursors have additional properties such as type, pressure and region of influence, and blobs can be described in various incremental messages that describe the bounding, contour and skeleton for example.
The other important thing that Martin is doing - and the reason for the wait - is to synchronize implementation of TUIO in other key libraries and clients. That is helping keep TUIO a standard for this kind of work. It’s not even really a full protocol - part of the beauty of it is that it builds on OSC.
If this isn’t quite making sense yet, stay tuned and we’ll show some of the specific applications and get you started with your own projects.
There has been a long tradition in live visuals and motion graphics, inherited from many other media, of maintaining a “secret sauce,” or the guarded formula of eleven herbs and spices. Ironically, for all you hear today “DIY” and “open source” in the same sentence, a lot of the motivation for doing something yourself has historically been doing something no one else can. Keep your secrets, and raise your value.
As our friend Bryant Place / CyberPatrolUnit sends over this latest set of live clips from a recent gig, and I browse through the comments, and reflect on the conversations I had last week at OFFF and during and following my own talk there, though, I’m struck.
The world has changed. First off, the Internet isn’t really about secrets. Your value is almost in direct proportion to how much you can share. Connections are forged through links of mutual exchange and good will. It’s not just about sharing your output or getting fans (the MySpace model), but sharing with a network of enthusiasts, and fellow artists. Those are the people from whom you often get real support (artistic, technical, and personal), gigs – and inspiration. (Even if you hate 8-bit music, that community is a really amazing model: their work to support each other and advocate for the whole subgenre has been I think the single biggest ingredient in their viral success.)
The visualist community increasingly itches not only to improve the quality of their own individual work, but everyone around them. A lot of us are in a battle for the future of this whole medium. Some parts of the world are devoid of live visuals, while others have mass-produced club visuals filling the nightlife.
Before I get carried away, the video itself is just the latest from the ongoing Interface 27 series. It employs a touch interface to control abstract visual pictures formed from streams of particles.
The reason I’m pulling back into the larger question is that these visuals are enabled by a library for Processing, a library we’ve seen here previously, developed by Memo Atken:
In a word: yummy. Or maybe that’s “bouncy.” Memo Akten’s brilliant ofxMSAPhysics brings open-source physics programming to the C++-based OpenFrameWorks environment, an artist-friendly combination with an elegant API. Coding physics doesn’t require an advanced degree. See today’s post on the beautiful My Secret Heart to watch this library in eye-popping action.
particles
springs
attractions (+ve or -ve)
collision
replay saving and load from disk (temporarily disabled in current alpha release)
custom particles (extend ofxMSAParticle and add to the system)
custom constraints (extend ofxMSAConstraint and add to the system)
custom force fields (extend ofxMSAParticleUpdater and add to the system)
custom drawing (extend ofxMSAParticleDrawer and add to the system)
By the way, don’t assume that the C++ folks get to have all the fun. Aside from the relatively simple but fast-and-useful traer.physics which inspired this more sophisticated outing, Java has stuff like:
There’s also a java.net project to bring the Open Dynamics Engine to Java, though I’m not sure that that’s terribly current. Java folks, feel free to chime in. But then, if you’re looking to migrate from Processing to OpenFrameWorks, this looks tasty indeed.
Let’s back up for a second. You hear terms like “spatially augmented reality” or “projection mapping,” and it may seem as though you’ve wound up in the latest movie adaptation of a Philip Dick sci fi short story. When we talk about these terms, we really mean one thing: getting projection off of flat, rectangular surfaces on walls and onto other stuff. It’s escaping the Drive-In Movie effect and layering digital images on top of the real world.
Remapping the projection image requires some number crunching. But, in case you hadn’t noticed, you have a supercomputer-powerful number cruncher sitting right next to you in the form of your computer. So what everyone is waiting on now is smarter toolkits for making projection more possible.
Brett Jones, a Masters Candidate in the Department of Science at the University of Illinois Urbana-Champaign, writes to describe his latest project.
First, I wanted to say that I am an avid follower of your blog and greatly appreciate the time it must take be on the forefront of the video projection mapping blogosphere.
I wanted to let you know about a post that you might find particularly interesting.
I am currently working on a spatially augmented reality/projection mapping toolkit at the University of Illinois at Urbana-Champaign. We recently posted two videos that might be of interest.
The first video is a work in progress demo of our toolkit and the second video is a new media installation piece that demonstrates the complex mappings that are possible.
What’s especially cool about this technique is that tracking uses embedded light sensors. A game engine can then map very accurately onto boxes and even mannequin heads.
It’s good stuff, and given how much mileage we’ve gotten out of walls by being creative with content, imagine how much more you could get out of additional projection surfaces - now with easier, accurate three-dimensional mappings?
We’ll be following this one, for sure, especially once the toolkit is ready for public consumption. Thanks, Brett!
1
