Add Max patch effects to DNA. from Livid Instruments on Vimeo.

Yesterday, we saw some splashy video distortion techniques applied to real-time video. You know what that means: it’s time to use these in live performance.

Liquidify Video, Live: Optical Flow GLSL Datamosh Technique

Here’s one start.Peter Nyboer, Max whiz and Livid developer, has run with the idea of squishing around video using optical flow analysis, and shows you how to add the effect to Livid’s Cell DNA VJ app. For Jitter users, this means you can rely on Cell for quick access to video taps and files, while adding unusual effects built in Jitter to get your custom processing on, not only with this example but any other patches you’ve created. One little detail of Cell DNA I missed – it requires Max 4 patches, not Max 5 patches. Peter has also posted a tutorial for working with that, after the jump.

And yes, if none of this is really making sense to you, you can go download the files and just try it out – no need to fully grasp all of the internals straight away.

Don’t want to use Jitter and/or Cell? The guts of Andrew Benson’s video datasplooshing technique is an OpenGL (GLSL) shader, so it doesn’t even rely on Jitter – Jitter can just be a convenient environment for playing around with such things. There’s word we may see a Quartz Composer wrapper around this shader, which would make it easy to use with software like VDMX.

Oh, by the way, I’m officially rescinding my editorial ban on the term “datamoshing.” Why? Because it means absolutely nothing, and therefore can be declared reasonably harmless. Also, unlike the term “glitch,” it comes without any baggage. We therefore have a nice, nonsense term for making video all mushy and unpredictable – a good thing.

motion distortion 2 from andrew benson on Vimeo.

Datamosh? (The “forbidden” but harmlessly meaningless word?) Video squishification? Mushy data?

Call it what you will, but applying real-time distortion and displacement to video so that video textures become flowing layers of pixels looks absolutely beautiful. Andrew Benson of Cycling ‘74 has only just begun playing with this in Jitter using GLSL shaders, and already the results are really compelling. (For a simpler example that looks more like the compression artifact technique we’ve seen recently, have a look at the second video – though, personally, I like the more sophisticated, layered approach of the video at top. This is going some very cool places.)

This is a Jitter patch, but would be simple enough to port to code for Processing, FreeFrameGL (which implements shader code), or other tools, too, in case you can’t bear being away from your moshness.

Andrew writes:

Earlier this week, I talked about some of the consumer-oriented feature improvements in Adobe Creative Suite CS4. But the most interesting development from Adobe is actually more on the developer side. Pixel Bender is a new format for making GPU-accelerated (and CPU multi-threaded) filters, which means fast blends (with different blend modes) and special effects. You can even use it for math operations. Now, whether or not you yourself are able to code these sorts of things, I’m sure we’ll see plenty of filters being passed around online. True, it means dealing with yet another way of creating these filters, but since it operates at a low level, it shouldn’t be too tough to adapt GLSL and other GPU code and techniques for Pixel Bender.

This should be particularly interesting for the open-source, multi-platform, Flash-based VJ tool Onyx, and for adding new, custom effects to your motion graphics arsenal. The filters will run natively in Flash 10 and After Effects CS4, with a plug-in coming to Photoshop.

Kevin Goldsmith at Adobe has been talking a lot about the new release on his blog.

Here’s an especially juicy tidbit from Kevin:

In addition to Pixel Bender Kernels (pbk) and Pixel Bender Bytecode (pbj), there is another Pixel Bender format. Pixel Bender Graphs (pbg). Pixel Bender Graphs are supported directly in After Effects CS4 and the Photoshop Pixel Bender extension. Pixel Bender Graph is an xml-based format that allows you to combine a network of Pixel Bender Kernels into a single effect for WAAAY more powerful filters. We’ll be posting a new version of the toolkit soon that supports the editing of these graphs and we’ll also be posting a specification for them soon on Adobe labs.

They’ve also made available a developer toolkit and even command-line tool. Updated: Kevin notes in comments here:

The Pixel Bender Toolkit has been available on Adobe Labs for almost a year now and anyone can install it and play with the language.

Also, there is a Pixel Bender developer exchange on adobe.com if you are looking for some sample code (although there is tons of stuff now on folks’ blogs).

Of course, that’s what we’ve seen with ActionScript in the past — developers get onboard well in advance of a release, thanks to publicly-available tools, code, and documentation, and then by the time the tool is out a lot of the discussion and examples are coming from developer and not just Adobe. There’s a lesson here for other framework developers (Apple, Sun, others, I’m looking at you).

Gotoandlearn already has a video tutorial on Pixel Bender. (auto-playing QuickTime link)

It’s not entirely revolutionary in that a lot of this does go on behind the scenes in graphics software. What is a big leap is making that functionality accessible to an end user of this kind of software. I think it should also put the pressure on Sun to expose more of this kind of functionality in JavaFX, because it’s clear people want it — and my sense is, like some of the other developer accessibility in CS4, this probably came from popular demand.

Who wants to again declare the GPU is dead? Not … yet, anyway.

For another excellent round-up, see John Nack on Adobe

He notes that Pixelero is already cranking out sample code. (Some of those examples inspire me to try some new techniques in Processing.) Never underestimate the size and passion of the Adobe community, that’s for sure.

Neon is among the best of a breed of obscure, indie VJ software gems around the world. The creation of mac/xplsv.com and later shine/xplsv.com, it’s made a mark not only as a VJ performance app but as a creator of real-time motion demos. It can event manipulate 3D Studio Max 7 scenes in real-time. All of this goodness could simply die as the developer moves on, but instead shine has decided to open source the code. That gives this Windows-only software a shot at a port to Mac and/or Linux, and turns it into a playground for would-be visual programmers, even if they just want to dabble in filter creation. And it gives the rest of us a lovely tool to add to our toolbox, free.

The feature set is pretty tasty:

• Layered effects: 10 effects layers (called FX instead of layers, oddly), with various blend modes, and a master effects channel - a really nice, practical architecture
• Real-time 3D with 3DS Max support, pixel shaders, vertex shaders (all apparently on DirectX, so some work would be needed for Linux, Mac, and OpenGL)
• Flexible formats: DirectShow, ffmpeg codec support, image support, SWF, live video capture
• Endless filters: Included filters, custom filters using pixel shaders, and a filter SDK with Virtual C++ examples. FreeFrame support, too, though not FFGL (yet) - Resolume has the win for FFGL support so far
• Live control, beat sync: The “beat manager” is the most insane part of this — and why this might find its way into your toolset. You have elaborate control of how things are synced to the beat, with even DJ-style pitch bend controls and per-element sync of parameters in your project. There’s also FFT (sound-reactive input) control, and MIDI.

Programmers can have at the SVN repository right away. Non-programmers get something special for free. I doubt this will shake your loyalty for your existing performance tool of choice, as a visual creation tool you’ll want to check out some of the cool 3D demos done, like this award-winning xplsv.com video, “Sound Pressure.”
(Edit: Auto-playing video moved after the jump - Jaymis)

Pd, aka Pure Data, is the free and open-source cousin of Max/MSP/Jitter. It’s powerful — even sometimes having technical advantages over Max — but has suffered from complex installation and dependencies, poor documentation, and an unpolished interface. Enter Pd-extended, a distribution that fills in those gaps. Pd-extended’s maintainer Hans-Christoph writes up what this is all about in an introduction on Create Digital Music, friendly even if you’re new to the Pd world.

Pd, Max’s Free Cousin, Gets Polish and Ease in Extended Build

Visualists should be especially interested in this latest release, because it offers much-improved out-of-the-box support for custom-patched 3D and video — especially if you’re on a Mac, for the PiDiP (though there are Windows and Linux improvements, too, and GEM works even with Windows).

Mac OS improvements:

• Image and video-processing PDP/PiDiP work out-of-box
• Anti-aliasing of boxes and lines in the interface
• New, purty icon

Linux:

• A .deb package for Debian and Ubuntu, with GNOME menu support. (`Bout time! Wonder if this means we’ll see it in the big Ubuntu repositories?)
• New icon

All platforms:

• GEM, the quasi-Jitter-like 3D and pixel library, has working shader support. Ed.: Truly outrageous.
• New libraries: mapping, msd, mrpeach net/OSC, flib
• [comport] is robust on all platforms (can you say Arduino?)
• Font-face and -font-weight command line options
• New font and layout is the exact same size on all platforms to the pixel. (previously you’d see some serious cross-platform glitching)

Pd Extended Release

And lest you have a bad taste in your mouth from the fugly older releases of Pd, Anton (vade) sends along the picture at the top of this story, showing the new UI from the Pd 0.40 dailies running on Mac OS X. Anton is also working on porting some of his brilliant visual patching from Max to Pd — and he’s living proof that even a Max die-hard can find at least some use with Pd, too. (The two environments are really, really close — sometimes confusion switching between them is because they’re so close, the differences can be confusing.)