Michael Faulkner of D-FUSE says the most interesting thing in this video: it’s when technology becomes redundant that it’s accepted as art. Photography gets invented, and suddenly painting - a business and a craft for centuries - is “high art.” (Don’t ask, incidentally, about what late 19th Century art critics and salon organizers thought was great painting. It was utterly dreadful.)
Ctrl Alt Shift is a 10-minute documentary, pointed out by Resolume developer Bart on their forums, featuring live visualists and audiovisualists. It has a number of things going for it: a terrific artist lineup, asking the tough, obvious questions about why fundamentally we do this stuff, and an editing style that makes projection, live-style edits, and eye candy animation part of the documentary object. It’s a tasty treat to watch if you know the artists, and even if the chatter in this video is the sort of thing you discuss over beers and V4s with your mates, it could be an ideal video to pass along to your friends who don’t yet entirely get what this whole thing is about.
Featured artists:
The Light Surgeons, D-Fuse, Hexstatic, Vj Anyone, Addictive TV, Vector Meldrew, and Fatamorgana
It’s a production by Dean G Moore and Simon Lane.
I know of at least one other documentary project on live visuals evolving. Of course, maybe what we need - given the imperfect and evolving nature of the medium - is more along the lines of a 24-hour visual network. I nominate … all of us.
Deride the mouse as you will. When it comes to two-dimensional control, the device is pretty amazing. It’s a reasonable way of amplifying small hand gestures into bigger gestures on a 2D plane. But what about 3D interfaces? Suddenly, the mouse becomes like playing charades, telling someone else what to do in a universe with an extra dimension.
What you need is interfaces that make sense in 3D. Some of these interfaces are out there; the missing link has been intelligent connections to software. To see how powerful this can be, look no further than Camera GripTools, a motion capture system for Maxon’s Cinema 4D modeling tool.
It works with a variety of devices:
Behringer BCF2000 MIDI fader panel (already a popular controller among VJs/visualists)
Nintendo’s Wiimote
Joysticks
Track IR 4 Pro is a head-mounted tracker for head movements. Pricing starts at just US$120, so this is absolutely a solution for mortals, and it appears to work really well – with 120 FPS tracking.
Polhemus Patriot VR Tracker – I’d never seen this before, and it looks utterly brilliant. It’s a six degrees of freedom tracker / 3D digitizer, made up of a small sensor. And it’s only £ 1,985 … oh. Okay, never mind.
For Cinema 4D users, this looks absolutely invaluable. It supports XPresso, CInema 4D’s fantastic modular, visual programming environment, with drag-and-drop support for objects. The free demo version is already pretty usable, with more powerful versions running EUR99-499 (though sadly you need the EUR499 version for full hardware support).
But I think the bigger message is how controllable this makes the 3D environment. This could be fantastic in an open source environment like Blender, or for live control in Blender, visual tools like VDMX, and custom creations in vvvv, Jitter, Processing, and the like. And while it may not be possible to make a system as accurate as the Patriot, looking at what they’re doing I suspect it should be possible to do a “ghetto” version on the cheap. (You know I’m all about that.)
2009 will be remembered as the E3 game event that embraced computer vision. Far from me-too answers to the Wii’s gestural controllers, we saw remarkably different visions of how computer tracking might work.
As expected, Sony had their own motion tracking system to unveil at their press conference. But unlike Microsoft’s 3D camera, Sony opted to build on their already-lovable PlayStation 3 Eye camera with wands with spheres. The controllers look ridiculous, and lack the magic of the Microsoft demos. But don’t dismiss them out of hand. (Sorry, there’s no way to write this story without lots of abstract puns.)
Much of what Microsoft showed was “conceptual” video – and some of the hands-on demonstrations had noticeable latency problems. Sony’s approach, meanwhile, was really quite literal in its demonstation. The tracking looks extremely accurate in 3D space, and latency appears to be minimal.
Above: Video of the press conference – check out how quick and accurate the tracking looks
The other good news for people working as artists and not necessarily mass-market game developers is that you can start to play with these ideas right now. Whereas Microsoft seems to have “lost” the once publicly-available 3D camera SDK for their solution, Sony is using an off-the-shelf camera you can buy right now and doing the rest of the work in software. I really like the use of tangible interfaces with cameras, because you can get more predictable tracking results, and you get the tactile feedback of having something in your hands. (I’m not sure I’d be as excited as they are about having a glowing ball on the end, but maybe I need to channel my inner raver.)
Anyway, here’s my humble prediction: it doesn’t matter how cool the demo looks or what sweeping statements anyone makes. Gameplay alone matters, and that means that what has to happen next is dependent entirely on the tracking working reliably and quickly, and developers building smart stuff around it that works as games. The same, naturally, is true for anyone doing broader interaction design and live visuals.
Sony is also getting further into the augmented reality arena. They have a Tamigotchi/Nintendogs-style augmented reality pet simulator, EyePet, for the console (see Joystiq’s hands-on), plus Invizimals, an augmented reality title for the PS3. Of the two, Invizimals is the most interesting. It’s funny that they immediately design it for kids (too bad, as I can see some office antics with this sort of thing). It’s also evident just how hard designing an effective augmented reality game can be. I don’t think skepticism would be wildly out of place – it’s clear that there’s something powerful about the concept, but not clear just what it will be.
And I don’t need to remind you, if you haven’t joined our tangible interface virtual party Saturday, head to http://hackday.noisepages.com/ – ARToolkit augmented reality is very much on the plate of stuff we’d like to see people play with. (The other schemes we’re using, Trackmate and reacTIVision, are better suited to 2D tracking on a surface, though they’re very, very reliable for that task.)
Artist Lia has created her first piece of art for the iPhone and iPod touch, something called PhiLia 01. It’s a quirky, gorgeous generative sound and visual app activated by movement, one that encourages users to save their own artwork.
Composer Morton Subotnik used to talk really eloquently back in the days of multimedia CD-ROMs and The Voyager Company about computers as “chamber music” environments. Instead of seeing the personal scale of technology as an impediment, he viewed it as something intimate and wonderful. So it’s fantastic to see artists engage mobile platforms as a way to have that relationship with a participant.
There’s now also a page up that is beginning to collect some of these particular artworks, focusing on generative-style interactive creations, and featuring work by our friend Memo Akten. Joshua Davis’ kaleidoscopic artmaking tool Reflect, which he showed for the first time at OFFF earlier this month in Lisbon, is enroute.
The way in which these tools are being created is interesting, too. PhiLia is built in OpenFrameworks, the open source C++-based development tool made friendlier for artists with integrated toolsets, a community of friendly creative folk, and simplified, speedy syntax similar to Processing. OpenFrameworks, thanks to its open source nature, has made its way onto the iPhone.
Part of what this demonstrates is that, while the iPhone itself is proprietary, some of the power of open source can still triumph. And, indeed, by basing work on this open source foundation, these same artists aren’t imprisoned by a single platform. PhiLia could be a desktop app, or on other mobile platforms once they support OpenFrameworks.
And, yes, it means I’m aching that much harder to get OpenFrameworks and/or Processing onto Android – it should be possible. (Java on Android is not identical to Java on desktop, so it can’t be a direct port – you can’t just install Processing on Android – but it is possible.) There are also still some wrinkles in the App Store approval process; it really is refreshing on Android (and presumably things like Palm WebOS) not to have those restrictions.
Then again, that’s the whole point: OS and specific platform shouldn’t have to matter, and open source software – and artwork – can be just as brilliant on a proprietary platform as an open one.
You can thank Lee Byron, Memo Akten, Damian Stewart, Zach Gage, and the core OF team (Zach, Theo, and Arturo). The “power of open source” is not some sort of magical whirlwind that surrounds code and makes things appear spontaneously – it’s blood, sweat, and tears (unpaid!) by real people. Although, if you get those real people together in a room and do some sort of battle shout or Care Bear Stare (sorry, I’m an 80s kid), it might help psych you up.
Microsoft’s Project Natal unveiling for Xbox 360 was no question a blockbuster of technology presentations, nothing short of sheer magic in a games industry that has lately looked somewhat backward-looking. The combination of a 3D-capable camera with facial and object recognition and vocal recognition and mic interaction takes already-smart elements and puts them together into something bigger. But demos are just that – it’s the reality of what’s happening in interaction design that’s interesting.
So, some more details on Project Natal:
Latency?
Note that the video in the post yesterday carries a significant disclaimer: it’s essentially a conceptual mockup, not a real demo. In videos we’ve seen of the current prototype, there does seem to be a significant lag between an action and its representation on the screen. This may have to do with the sheer amount of data and analysis that’s being done on it. Unfortunately, as this is only in prototype stage, it’s impossible to do much more than speculate.
I’m not the only one to notice this: Keith Lang, interaction designer at Plasq, sees the same concern in his (excellent) round-up of coverage of Project Natal:
Don’t underestimate how important the latency could be, either. Even tiny differences in latency can have a major impact on how someone feels about an interaction. This is also significant to music people, who generally like their interactions to use tiny latencies and approximate the rate of the audio they’re controlling.
I’ll reserve judgment until the final version, naturally! But it’s something to watch.
Johnny Chung Lee and the 3D Technology
The ingenious creator of various Wii tracking hacks, it seems, is now with Microsoft. (Nintendo, your loss. Rest of the world, he has code tools on his site, so even without hiring the guy, you can benefit from his knowledge.) Cristian Campo spots the news in our comments.
For his part, Johnny is careful to note that he’s not responsible for what you see, but is working with them on productization.
He can’t reveal anything but what’s public, but he does have some more extensive details on the technique – essentially, information that is public but in a more technically-specific form:
The 3D sensor itself is a pretty incredible piece of equipment providing detailed 3D information about the environment similar to very expensive laser range finding systems but at a tiny fraction of the cost. Depth cameras provide you with a point cloud of the surface of objects that is fairly insensitive to various lighting conditions allowing you to do things that are simply impossible with a normal camera. But once you have the 3D information, you then have to interpret that cloud of points as "people". This is where the researcher jaws stay dropped. The human tracking algorithms that the teams have developed are well ahead of the state of the art in computer vision research. The sophistication and performance of the algorithms rival or exceed anything that I’ve seen in academic research, never mind a consumer product. At times, working on this project has felt like a miniature “Manhattan project” with developers and researchers from around the world to coming together to make this happen.
We would all love to one day have our own personal holodeck. This is a pretty measurable step in that direction.
Creepy Kids
Seaman, you’ve got nothing on this. (Sorry, Leonard Nimoy.)
Yes, it seems Peter Molyneux’s latest project uses Project Natal to simulate interactions with a kid. This does start to make me wonder if – as “realityengager” wonders in CDM comments – we should just go out into the real world and interact with that. (Daddy? Why won’t you play with me any more? Why are you only playing with Xbox 360 Milo kid?) But as a tech demo, of course, it’s mind-boggling – and it’s nice to think what it might mean for storytelling.
See the video at top. Molyneux suggesting that even science fiction hasn’t written about this sort of technology is especially absurd, as it seems science fiction spends most of its time writing about exactly this, but you get the point.
I just want Project Natal support in XNA so artists can play with this stuff. Hear us, Microsoft?
5
