The pixel-mangling madness continues. Here are the latest goodies to show up in our inbox.

Enrico Nau aka naus3ayt is playing with code-for-artists environment Processing (site | cdmo tag). It’s yet another example that reveals that computer vision, aside from the useful-ish applications like tracking what you’re doing, can be used to purely aesthetic ends. No code excerpt here, but the best place to begin here would be the official Processing tutorials or the OpenCV tutorial recently posted here on CDM.

Blog post here:

Chroma-sampler: painting with light

Enrico describes his creation to us:

I recently coded a toy that samples colors from a video feed and used them to create virtual brushes; similar mood to data moshing, different approach.

In other news, we have this gorgeous glitchy … something or other. Okay, actually, I have no idea what this is or where it came from (and neither does tipster con), but boy, does it take the opposite tack compared to the one suggested by David O’Reilly – it’s a maximalist, intentional overuse of a particular data compression artifact. In this case, though, it fits nicely with the musical content, which is what O’Reilly suggested.

godxilliary.com [QuickTime link]

I guess my ban on datamosh, incidentally, is well over, especially after I temporarily moshed the story database on Create Digital Music by importing with some setting wrong. (It doesn’t help that WordPress and MySQL once defaulted to Swedish. Swedish: the new lingua franca.)

Anyway, enjoy – and keep experimenting, everyone. Experiments, by definition, can be ugly, and that’s good. It’s all part of the process.

Jitter works brilliantly when it comes to processing signal - and that means for signal-like work with video and textures, it’s fantastic, as well as the usual Max-y tasks like processing input from physical sensors and input devices and the like. But try to do a whole lot of sophisticated 3D work, and Jitter may not be the best tool. For game-style 3D graphics and interaction, you want some standardized rendering and scene graph tools to take care of the hard work, plus physics and other capabilities that bring together your 3D scene.

That’s why [myu], the Max - Unity Interoperability Toolkit, looks so appealing. It not only allows for bi-directional data integration (via TCP) of Max and the Unity game engine, but can dynamically pass textures between the two. For those of you comfortable patching, say, chains of shader processors in Jitter, that means you can very quickly add some of the tasty 3D scene powers of Unity. Put together your textures in Jitter, and, say, dynamically process input from a Wii Fit balance board, then bring the input data and textures into Unity. (Unity is a friendly, elegant game engine built in C# and Novell’s open-source Mono implementation of Microsoft’s .net. Unity had previously been Mac-only but with a major new release now runs on Mac and Windows.)

The toolkit is the result of research at Virginia Tech Interactive Sound & Intermedia Studio director Dr. Ivica Ico Bukvic.

Needless to say, this could have powerful implications for all kinds of live and interactive installation applications. And yes, it is all released under the GPL.

[myu] Max-Unity Interoperability Toolkit v.1.0 Released [Cycling '74 Forum]

More Max+Unity Game Engine Goodness, with Powerful Toolkit for Max, Jitter, Pd

Teaching Adaptive Music with Games: Unity + Max/MSP, Meet Space Invaders!

For other examples of combining Max and Unity - in this case for Max’s musical powers and Unity’s gaming prowess - see another story from today:

Teaching Adaptive Music with Games: Unity + Max/MSP, Meet Space Invaders! [Create Digital Music]

Updated: About those textures…

Ico follows up to answer our questions about how you might use textures with Jitter and the Unity Game Engine, via his [myu] toolkit:

motion distortion 2 from andrew benson on Vimeo.

Datamosh? (The “forbidden” but harmlessly meaningless word?) Video squishification? Mushy data?

Call it what you will, but applying real-time distortion and displacement to video so that video textures become flowing layers of pixels looks absolutely beautiful. Andrew Benson of Cycling ‘74 has only just begun playing with this in Jitter using GLSL shaders, and already the results are really compelling. (For a simpler example that looks more like the compression artifact technique we’ve seen recently, have a look at the second video – though, personally, I like the more sophisticated, layered approach of the video at top. This is going some very cool places.)

This is a Jitter patch, but would be simple enough to port to code for Processing, FreeFrameGL (which implements shader code), or other tools, too, in case you can’t bear being away from your moshness.

Andrew writes:

Part of the big idea behind Resolume Avenue 3 is putting audio and video in the same real-time performance app, without sacrificing quality or live processing in either. One key ingredient, hope the developers, is a new codec that transfers video decoding to the GPU without taxing the resources the GPU needs for accelerated effects. For lower-resolution footage, you may not notice a difference, but with a modern graphics card, Resolume claim intense multi-layered performance at higher HD resolutions. We’re keen to see just how that stacks up in benchmarks, not only Resolume vs. Resolume, but Resolume’s DXV up against codecs like Photo JPEG in other apps. But before we get there, we first wanted the inside scoop from the developers.

I got to speak to Resolume’s Bart van der Ploeg. I knew Bart had been dying to show us the new codec for months, so here I get to grill him about questions a lot of us have.

The good news: no, this won’t screw up your GPU-based shaders.

How did DXV come about?

While doing our research for a new GPU-based rendering engine for Resolume back in 2006, we discovered that a large bottleneck for a fast GPU video engine was the limited bandwidth to the video card. The rendering of video frames with fancy effects at a high resolution was absolutely no problem for the GPU, but half of the time it was just waiting for all the video frames to arrive to the video card’s VRAM. We just could not get the video frames to the video card fast enough to feed the GPU to its maximum potential.

We reduced the amount of data that has to travel to the video card by feeding it compressed video frames instead of uncompressed frames that come from any other traditional codec. This makes the whole engine run much faster and reduces the CPU usage so we can use that for other things like audio effects.

Whatever your feelings about Microsoft, their R&D operation employs some of the smartest minds in the business - and has done so for a long time. TechFest is the event that shares some of the best experiments. Some turn into products, others don’t, but the events always have some gems, and can provide plenty of inspiration.

What I’m personally most excited about is the pace of progress in image analysis and processing. You’ve already seen me marvel at projects that merge and stretch images in 2D and 3D, seamlessly, and each time, I wondered about video:

Microsoft Live Labs Photosynth Transforms 2D Photos into 3D Worlds; Hands-on

Savvy Stretching: Free Pixel-Resizing Tools, But What About Real-Time, Video?

Well, video has arrived, as pictured at top:

Real-Time Stitching of Mobile-Generated Videos

The ubiquity of mobile phones in today’s society enables the capture of multiple videos of the same event, each with a small field of view. This Microsoft Research technology unveiled during TechFest 2009 enables those small videos to be stitched together into one higher-resolution video in real time.

The project is the work of Ayman Kaheel, Motaz El-Saban, Mahmoud Refaat, and Ahmed Abdul Hamid of the Mobile Multimedia Content Services Group at the Microsoft Innovation Lab, Cairo. That’s proof that great R&D can and will happen in cities that aren’t called Cambridge (Massachusetts or England).

Microsoft, if you’re listening, I’d love to see this as a developer framework. I’m sure there are uses for end-user projects, but imagine how many other ideas couldn’t be realized by any single developer. Live visualists could do amazing things with this, stitching together bunches of different cameras.

In another visualist-ready achievement, Microsoft is projecting onto a really big dome with a single projector, thanks to a wide-angle lens, as seen below. It’s a simple project, but given the complex, multi-projector setups I’ve seen doing the same thing, it’s a very useful project. (Now here’s a case in which I wish they had done more documentation, as this is probably not something they’re going to patent.)

I have some contacts at Microsoft Research, so I’ll see if I can’t find out more.