tbeta preview from ~ on Vimeo.

Look out: multi-touch has a new rising star. The tbeta library (short, oddly, for The Beta) is an open-source framework for computer vision and multi-touch, and it’s particularly good at following your fingers. It’s a descendent of touchlib, with some of its ideas, though a completely new code base. tbeta now powers the finger tracking routines in the BricK multi-touch table (below), at which we take a detailed look today on Create Digital Music:

Spaces and Roots: Manipulating Sound with Processing + Touch, Tangible Interfaces

Audience from Chris O’Shea on Vimeo.

Chris O’Shea and rAndom International have completed a lovely installation at the Royal Opera House (UK). 64 mirrors move, each distinctively, to follow moving attendees who catch their “attention.” The installation is powered by Chris’ custom code and rAndom’s hardware and circuits, build on C++, OpenFrameWorks, and Intel’s ubiquitous open source computer vision library OpenCV.

I really enjoy how elegant the resulting design is, and the way it fragments the faces of viewers in a sea of mirrors, bobbing around with simulated intelligence.

Audience for Deloitte Ignite Festival [Project Page, Chris O’Shea]

I expect this could inspire other computer vision projects, or motorized screen concepts in place of mirrors.

Here’s what the video analysis software interface looks like:

See also: other photos of Audience on Flickr

The VJ and live visualist of the future isn’t just about DJ metaphors and what happens in clubs. It’s about a convergence of new interface technologies for dealing with visual material in a more fluid, flexible way. It’ll change not only visual performance, but how we express ourselves in digital visuals, as well — something we’ve already seen happen with non-linear video editing and vector and bitmap graphics software, but taken further.

Vade points us to a couple of glimpses of technologies being researched now that will help enable these changes.

Chris O’Shea (also of the blog Pixelsumo) has a brilliant installation that allows people to see through walls. It’s an idea I’ve seen done before, but Chris actually makes the effect convincing, by giving visitors an infrared torch (what we’d call a flashlight here in the States, though torch in this case is an even better word). Software tracks the position of the IR emitter via an overhead security camera, and the whole thing is coded to make the impact realistic.

Software is coded in OpenCV (an open-source computer vision library from Intel, in C++) and OpenFrameworks (a lightweight multimedia C++ framework for artists, on some level trying to do for C++ what Processing has done for Java).

There is a childlike quality about wanting the ability to see through walls with x-ray vision like a superhero character. This memory is something Chris O’Shea wants to capture in the interactive installation Out of Bounds. The work encourages visitors to bore through the walls of the museum and engage in a ‘behind the scenes’ experience with an x-ray torch. This playful interaction encourages childlike curiosity in young and old alike, and opens up a portal into the Museum’s forbidden spaces.

Shine the torch at the wall to reveal the secrets hidden beneath. Pay an anonymous visit to the staff office, collection’s store, workshop, roof hatch or plant room.

Out of Bounds on Pixelsumo
Designers in Residence Program, Design Museum, London
Now showing at DesignTide, Tokyo
Out of Bounds Project Description, Documentation

Seeing the IR Rainbow

The results are so realistic, Chris says he’s gotten emails from people thinking it was real. But the whole effect is purely illusion; IR emitters of this type (near-infrared) aren’t capable of penetrating surfaces. That’s something I had to explain to people when I was using IR-sensitive cameras myself (nothing fancier than a DV cam in night vision mode). Higher up the IR spectrum, it is possible to sense heat through walls, though the effect is nothing like the Superman “X-ray vision” seen here.

But among those fooled by near-IR’s “magical” properties? The US government, evidently. It seems our government, in the latest expression of its infinite wisdom, has placed import/export bans on simple IR flashlights for security reasons. (This sounds really odd, even for us; if anyone knows more about this, I’m curious to know.) I’m not sure what the precise security threat would be; maybe commandeering someone’s TiVO by shining at its remote control receiver? Don’t tell the US government, but there’s all kinds of mobile lighting technology that allows you to see in the dark, too. Like Mag-Lites.

Reimagining Projection

What I most like about this project, separate from how lovely it is as installation art, is that it breaks up the projection itself. The spotlight mechanism is simple, but it suggests the potential for letting viewers control projection, and “virtualizing” the projected image rather than letting it simply be a rectangle. (Not that I don’t love rectangular images — I sure spent a lot of time watching TV episodes in rectangles this weekend. But you get the idea.) I’ll be interested to see how ideas like this show up in clubs and performances.

I’m boarding a Greyhound bus for Boston and the Ideas in Motion weekend of Boston Cyberarts, two packed days of events on dance, motion, interactive technology, artificial intelligence, and augmented bodies:

Boston Cyberarts Festival 2007 - Ideas in Motion: The Body’s Limit

If you’re in Boston, stop by and see us! I’ve been preparing two works — creating digital music and digital motion in each, if you will. Palinopsia at 2pm Saturday is my own work with choreographer Elise Knudson and dancer Pauliina Silvennoinen, for which I’ve done a music score and live visuals in Jitter (some infrared, as seen above). Take a Seat is Andrea Haenggi’s work in a conference room of the SHARE Swiss Consulate 11am Sunday, at which I’ll be mixing live sounds in Ableton Live and built an interactive barcode scanner that sends barcode-label messages to the dancers.

And of course the whole weekend is packed with dances and lectures and other good stuff. If you’re there, do introduce yourselves. If not, I hope to report back next week.

(I sometimes feel like I hit the body’s limit on a regular basis, but I’m a weakling who doesn’t respond well to stress.)

1080p, 3 CCDs, a couple of inches in size, and 2.3 ounces? Yep, you heard that right:

Iconix Products

The ICONIX HD-RH1 is a tiny HD camera, with output in SDI, DVI-D, or analog BNC, up to 1080p resolution, and apparently serious performance for “POV” applications. In fairness, only the head is this tiny; you have to attach to a larger controller to use. But for hiding a tiny HD camera in an installation, this is a true beauty. Now the bad news: US$15,000 sticker price, though rentals are available.

And they look cute as eggs:

Mmmm, HD omelet.

Imagine this rig: an HD camera for installation, simultaneously routed to a computer for analysis, mixed back to full HD visuals for content. Of course, for now, you’d likely want to downsample the resolution for doing motion detection and such — unless someone knows of a hardware solution that could assist a computer CPU in doing the computation. (I expect at some point, with more advanced GPUs, we’ll be doing more GPU-native computation of actual motion detection.) I could see tools like Processing/Java as wrappers for animation, while all the hard number-crunching happens somewhere else.

Anyway, that leaves you lots of time to write the grant for renting the camera.

Via the fabulous French-language blog Le Collagiste VJ, which is what I would aspire toward for CDMo if we wrote natively in French.

Looking for new projection surfaces, but too tired to get out of bed? Wait a minute — we may be onto something:

Now, admittedly, the IKEA installation doesn’t realize the full potential of this idea. (Hey, corporate clients … you know how it is.) I think bed projection is an idea worth stealing. How adding an interactive fortune cookie reader, appending, you know, “– in bed”? Even better, of course, would be the ability to lie in the bed without casting a shadow, which might require some extremely unorthodox back projection. (Erm, so to speak.)

Via the Netherlands-based Fresh Creation.

Processing.org’s fabulous exhibition has been updated with Grass, an installation by The Barbarian Group for Saturn (the car) shown at the recent Wired Nextfest here in New York. As usual, something that looks quite simple has a lot of elements:

1. Installation: Barbarian Group relied on veteran Obscura Digital for the installation; auto exhibition exhibits are an ancient, time-tested trade, and calibrating projectors perfectly is a b****. (I’ve been doing a lot of it this month. I’m terrible at it. I hate it. Straight lines and I don’t get along.) Four back-projected DLPs do the dirty work.
2. Camera tracking: This part is comparatively simple: the camera looks for dark spots against the projection (presumably in the visible light spectrum, since shadows is what you want) and compares frame-to-frame motion for the movement.
3. Grass: The grass is a series of segments, bent along a gradual decay. (Scroll down in the example above; they explain it fully.)
4. Perlin Noise wind: Hooray for Perlin Noise! This random generator works very well for this kind of organic material. (You can also generate Perlin Noise in Flash 8 and later.) Google it; it’s one of the greatest things ever.

The results are lovely, with interspersed text floating in. Of course, Saturn’s needs were comparatively modest; they just wanted a nice, pretty background for pushing their hybrids — it’d be nice to see these concepts taken further.

OpenGL in a Browser

Equally, interesting, though, is that the creators present a simplified version of their Processing file rendered in OpenGL rather than P3D, which is what you’d normally use for 3D in a browser with Processing. Yes, that’s an actual OpenGL rendering inside my Firefox window below. Your mileage may vary; Jaymis reports he can’t make it work in Opera.

If you’re interested in doing this yourself, java.net has an example using the JOGL Applet Launcher:

jogl-demos: JOGL Applet Test

I haven’t yet tried to hack Processing myself to make Processing files work, but both your Processing sketches and other Java JOGL projects should work. Note that you’ll want to look at the video, too, as they had to compromise a bit to make it work well in a browser. Viva Java — can’t do that with Flash, for sure. (Of course, I may have spoken too soon, as Firefox 2 RC3 promptly crashed after looking at the JOGL player.)

I’m guessing that a lot of the polish of the finished work has to do with using creative filtering and anti-aliasing in JOGL, as well.

flickr Feathers

While we’re on the subject of lovely, organic designs, blprnt has created a beautiful sketch that reads image color values from flickr using Flash 8 and later:

Plumage: FineFlickrFeathers (via Processing Blogs)

Flash, not Processing in this case, though you could do something similar with Processing. (And the above project could be adapted for things Flash does well — depends on how 3D you want to go, in part).

But let’s put the technology aside a moment: the real lesson here is that great work is created by working:

I’ve been somewhat disappointed with my creative output as of late. So, with a day off of client work, I set out this morning to make something interesting before the end of the day.

Yep, we’ve been there. Doing a one-day opus is a great idea. As for getting that day off? That I can’t help you with.

Learning Processing

“Is there a good, basic tutorial for Processing?”

This question gets asked all the time. Processing is a powerful, open source, cross-platform (Mac, Windows, Linux) tool capable of all kinds of multimedia magic from video processing to generative 2D and 3D art and animation. There’s no graphical UI, just simplified Java code, but it really is simple enough to be used by non-programmers — frankly, I find it much easier than ActionScript (Flash) or Lingo (Director). But download it, and it’ll be extremely unclear where to begin. There’s no step-by-step tutorial, just a reference to commands and some sample code.

Fortunately, Josh Nimoy — the NYU whiz who created the excellent JMyron computer vision library — has written just the kind of “Processing quick start” you need:

Processing Tutorial for Macromedia Minds

It’ll be particularly friendly if you come from Director or even Flash. There are also a couple of reference books due, one from Processing’s original creators and another, more practical reference, in the works for next year by NYU faculty Daniel Shiffman. Even after those books have come out, though, I expect people will still find a quick start useful. If you’re trying to learn Processing or even just want to see an overview of what it is and how it works, this is for you.

Incidentally, ignore the references to Proce55ing and P5; when Processing had zero Google rank for confusion with other forms of (lower-case) processing, this odd spelling was needed. Now, virtually no one uses these names.

Teaching Computers to See

Processing happens to be a great environment for working with computer vision applications, but that brings us to the next question: what’s computer vision, and how do you learn about that? Computer vision is a broad term applied to various techniques for teaching computers to “see” the way we do. This can be very simple, like basic motion detection and using frame differencing to display parts of an image that change from frame to frame. It can also be more sophisticated and interactive, from basic collision-detection with individual pixels to tracking discrete objects as they move around the screen.

Working with computer vision is a little like working with video in general: some of the basic techniques are simple and commonly-understood, but turning them into something artistic is easily a lifetime endeavor. That said, here’s one of the best places to start I’ve seen yet:

eating video: Workshop by Zachary Lieberman

Zach Lieberman is best known for his work on Messa di Voce, a beautiful interactive music piece with simple but elegant animations of circles and other shapes floating around performers and appearing to stream out of their mouths. For this workshop in Barcelona, he provides some terrific background reading (including the tutorial above), plus various code examples in Processing.

This is extremely basic stuff in the workshop examples, but it’s a good place to start. (Windows users will need WinVDIG as this uses the built-in Processing capture library and QuickTime for Java.) Next, I’d suggest turning to JMyron (which also saves Windows users the trouble of using WinVDIG); check out the examples included with that download:

Myron aka webcamxtra aka JMyron from Josh Nimoy

Read through these materials, work through Josh’s tutorial, and work through the examples in JMyron, and you’ll be well on your way to creating some basic interactive motion pieces in Processing. I’ve been developing some of my own, so since that last step isn’t terribly well-documented, stay glued to Create Digital Motion and I’ll try to write something up in the next couple of weeks.

One notable problem with Java (and thus Processing) is that, on its own, it’s not the best solution for doing the pixel-by-pixel processing and computer vision / motion tracking / video analysis, as it’s not as efficient for those kind of tasks as C/C++. The motion tracking / computer vision library for processing JMyron is actually written in C++ and wrapped in Java, for instance — though it’s important to note that the ability to wrap C/C++ code in Java is very nice. (And once you’ve done the basic analysis, you can get away with some iterating through pixels in Java/Processing, and it is fast for some related tasks like doing the actual animation.)

But C++ can be a bit unfriendly. There are libraries that are very helpful for doing this kind of work, but you have to track them down. Chris O’Shea of Pixelsumo writes about a new solution that could bring some of the basic ease of use of Processing’s integrated libraries to C++ work, in the form of Zach Lieberman’s new C++ library “OpenFrameWorks”:

OpenFrameWorks at Ars Electronica [Pixelsumo]

I’d keep a close eye on this, as Lieberman’s extensive experience building work that uses these techniques means he knows a lot about what’s needed in the way of performance. But there are some major caveats that mean this isn’t quite “Processing for C++”: