I’ve been blogging effectively every day for nearly five years, so it’s hard to avoid novelty - that’s kind of what we publish. The question is, is it the novelty that’s important, or do you see these as little steps toward something greater that hasn’t happened yet? I tend to favor the latter.
Here’s the question: can augmented reality - using computer vision tools to mix computer graphics with stuff in the real world - become a real medium and not just a gimmick?
If it works in a game, you begin to think there’s really something to an interactive design. I’m not certain yet about the upcoming PSP game from giant Sony. Georgia Tech researchers, on the other hand, are coming up with stuff I really want to play.
Oh, yeah - and it makes me crave some Skittles. (Product integration, anyone?)
Here’s a really elegantly-designed multi-touch table. It uses two really key pieces of open source technology: the Community Core Vision (“CCV”) tracker, formerly known as tbeta, and a lovely framework for coding visuals called PyMT.
PyMT, as the name implies, is a Python-based framework. I’ve gotten to know Nathanaël Lécaudé, a talented artist and coder who was nice enough to put me up a couple of nights while I was in Montreal; he’s one of several core coders. They’re doing a lot to really encapsulate functionality in widgets in a nice way. Features of PyMT include an event framework, specialized widgets for gesture, touch, and layout, and connections to OpenGL, OpenGL shaders, and sound. You can even work with the enduring, evergreen synthesis language Csound using its Python bridge, the oddly-named but powerful Ounk.
Of course, that’s all plumbing. It’s nice to see this applied to something powerful and educational. From the Vimeo description:
"Puddle of Life" is an educational multi-touch installation that was designed for Coimbra’s Science Museum (Darwin exhibit), demonstrating the theory of natural selection, as part of Darwin’s 200th anniversary.
The installation is composed of a round multi-touch table whose surface represents a virtual environment where 4 species of creatures live in. Each of these little creatures have different physical characteristics visually expressed by it’s quantity and/or size: Vision, Locomotion and Fur.
The player interacts with the game by touching the surface. He has to maintain full awareness of his creatures emotions (visually represented by a cartoon like ballon) and using this information to properly choose the mating partners on the control console. This console also allows the user to select the most appropriate descendent from 4 possible mutations, resulting from the reproduction.
The player’s objective is to help the species he controls achieve the highest number of creatures of its kind. Since this world suffers from climate change the player must assure that his creatures are well adapted to this ever-changing environment by choosing the best balance between mating partners and descendants.
The species move in a swarm, but they split when the predator is near. When a creature sees food it warns the nearby siblings and they all run towards it. Of course only the ones who have best locomotion reach it. The vision gives them the ability to see the predator sooner and the ability to see the food further ahead too. The fur is useful to them when the temperature is low but harmful when it’s hot, leading the creature to a shorter lifespan.
The player wins when they reach about 18 living creatures and loses when all of it’s creatures die from cold/hot, predation, famine or old age.
The project is the work of Tiago Serra and his collaborators, who have also been PyMT contributors (see comments). Great work, gang. We’ll be watching for more.
We have the tools. We have the techniques. Now, what happens when you put technology for tracking physical objects into the hands of artists around the world?
On June 6, members of the CDM community joined in our first “global hackday,” assembling tangible interfaces on tracking tables. Martin Kaltenbrunner of reacTIVision and the reacTable joined us from Vienna, Austria, while Adam Kumpf of the OpenFrameworks-powered Trackmate and MIT Tangible Media Group chatted from Cambridge, Massachusetts.
Much of the day was about saying “hello, world,” and helping each other through getting cameras working, troubleshooting, and the like. But there was an extraordinary variety of ideas even in one day. I can only imagine where things might go from here. I can also see the tools people were developing as being expressive for live visuals and visual performance - and via a really cheap interface, too.
Some of the accomplishments of our first day:
Endless ideas: Drawing interfaces using objects, a floor tom as a housing, a musical instrument with soda bottles as the interface, a game with blocks featuring the Tokyo skyline, and others.
Troubleshooting data: Both the Trackmate and reacTIVision projects got lots of feedback about how people were using the tools, where tracking was and wasn’t working, and where people got stuck up. We also compiled lots of information on cameras, drivers, builds, and operating systems. I’m working with Adam and Martin to dig through a lot of this information so we can compile a really practical guide to make it easier for people to create their own projects.
Special guests: Marco had his augmented magic show and we had the beginnings of an interactive glove. Check out the video highlights to see what the NYC event was like.
noisepages for networking, and other tools: Livestream video was a bit of a mixed bag; I’m still looking for easier ways of doing that (both on the video shooting side and the streaming side.) Text chat was an easy win, though IRC can still be cumbersome; I’m looking into integrating standardized XMPP chat instead, and providing access via any client or a webpage. But the other big success story was that noisepages worked nicely for documentation; see the fritzcrate and i3games sites for great examples!
Building Communities Around the World
A real highlight to me was getting to hear from Valeria (jalea.tv) and Jose (Estado Lateral Media Lab), visiting New York from Argentina. They talked about what the scene is like in Buenos Aires, and touched on issues of community, learning, open source, and the multilingual world of coding. And they do some really beautiful and hip visual work, both commercial and experimental. We also wound up with a significant amount of the online chat being in Spanish. That to me is a healthy sign - “global” really doesn’t necessarily translate to “English.”
Hackday Results, and the Future
For more documentation, head to our noisepages site:
And, yes, this is only the beginning. My suspicion is that a single weekend would be enough to get workable tracking projects going - especially as we iron out some technical wrinkles. But we’d certainly love to do more of this, whether it’s another “official” hackday or simple an open lab with chat and sharing.
noisepages is still in alpha state, but we’re working aggressively to move forward to beta, and content placed there is safe and future-proof. (Most importantly, I’ve fixed the jpeg library on the server so that avatars work!)
Deride the mouse as you will. When it comes to two-dimensional control, the device is pretty amazing. It’s a reasonable way of amplifying small hand gestures into bigger gestures on a 2D plane. But what about 3D interfaces? Suddenly, the mouse becomes like playing charades, telling someone else what to do in a universe with an extra dimension.
What you need is interfaces that make sense in 3D. Some of these interfaces are out there; the missing link has been intelligent connections to software. To see how powerful this can be, look no further than Camera GripTools, a motion capture system for Maxon’s Cinema 4D modeling tool.
It works with a variety of devices:
Behringer BCF2000 MIDI fader panel (already a popular controller among VJs/visualists)
Nintendo’s Wiimote
Joysticks
Track IR 4 Pro is a head-mounted tracker for head movements. Pricing starts at just US$120, so this is absolutely a solution for mortals, and it appears to work really well – with 120 FPS tracking.
Polhemus Patriot VR Tracker – I’d never seen this before, and it looks utterly brilliant. It’s a six degrees of freedom tracker / 3D digitizer, made up of a small sensor. And it’s only £ 1,985 … oh. Okay, never mind.
For Cinema 4D users, this looks absolutely invaluable. It supports XPresso, CInema 4D’s fantastic modular, visual programming environment, with drag-and-drop support for objects. The free demo version is already pretty usable, with more powerful versions running EUR99-499 (though sadly you need the EUR499 version for full hardware support).
But I think the bigger message is how controllable this makes the 3D environment. This could be fantastic in an open source environment like Blender, or for live control in Blender, visual tools like VDMX, and custom creations in vvvv, Jitter, Processing, and the like. And while it may not be possible to make a system as accurate as the Patriot, looking at what they’re doing I suspect it should be possible to do a “ghetto” version on the cheap. (You know I’m all about that.)
Shamus Young’s “Pixel City” feels like flying in a helicopter into the art from Ghost in the Shell, or discovering a metropolis inside your computer. The latest work from an undiscovered YouTube talent, the software itself will be released under an open source license. I don’t need to tell you this could inspire other experiments for urbanist visualists wanting to work with real-time landscapes.
It’s also interesting that the process itself becomes part of the artwork: it’s by understanding how each element is pieced together that you really connect to the meaning of the whole.
This is a demonstration of a program I wrote to generate and fly through a dynamically generated city. You can read the step-by-step of how it was made at my website:
2
