Add Max patch effects to DNA. from Livid Instruments on Vimeo.

Yesterday, we saw some splashy video distortion techniques applied to real-time video. You know what that means: it’s time to use these in live performance.

Liquidify Video, Live: Optical Flow GLSL Datamosh Technique

Here’s one start.Peter Nyboer, Max whiz and Livid developer, has run with the idea of squishing around video using optical flow analysis, and shows you how to add the effect to Livid’s Cell DNA VJ app. For Jitter users, this means you can rely on Cell for quick access to video taps and files, while adding unusual effects built in Jitter to get your custom processing on, not only with this example but any other patches you’ve created. One little detail of Cell DNA I missed – it requires Max 4 patches, not Max 5 patches. Peter has also posted a tutorial for working with that, after the jump.

And yes, if none of this is really making sense to you, you can go download the files and just try it out – no need to fully grasp all of the internals straight away.

Don’t want to use Jitter and/or Cell? The guts of Andrew Benson’s video datasplooshing technique is an OpenGL (GLSL) shader, so it doesn’t even rely on Jitter – Jitter can just be a convenient environment for playing around with such things. There’s word we may see a Quartz Composer wrapper around this shader, which would make it easy to use with software like VDMX.

Oh, by the way, I’m officially rescinding my editorial ban on the term “datamoshing.” Why? Because it means absolutely nothing, and therefore can be declared reasonably harmless. Also, unlike the term “glitch,” it comes without any baggage. We therefore have a nice, nonsense term for making video all mushy and unpredictable – a good thing.

motion distortion 2 from andrew benson on Vimeo.

Datamosh? (The “forbidden” but harmlessly meaningless word?) Video squishification? Mushy data?

Call it what you will, but applying real-time distortion and displacement to video so that video textures become flowing layers of pixels looks absolutely beautiful. Andrew Benson of Cycling ‘74 has only just begun playing with this in Jitter using GLSL shaders, and already the results are really compelling. (For a simpler example that looks more like the compression artifact technique we’ve seen recently, have a look at the second video – though, personally, I like the more sophisticated, layered approach of the video at top. This is going some very cool places.)

This is a Jitter patch, but would be simple enough to port to code for Processing, FreeFrameGL (which implements shader code), or other tools, too, in case you can’t bear being away from your moshness.

Andrew writes:

Shaders, snippets of code for processing pixels and 3D points on your 3D card’s GPU, are cool — that much you may know. You may even know that you can use shaders — designed for 3D applications — to perform powerful video-processing tricks, as well, at high speeds, even on a relatively lowly laptop. How to actually build your own — that may be elusive. So, at long last, Cycling ‘74 has published a great, beginner-friendly (even for non-programmers) tutorial on building your own shaders:

Your First Shader

The author is Andrew Benson, who is my hero as far as coming up with great Jitter examples. (Every time I’m looking for some model for a technique I have in mind, I keep stumbling on his sample patches in the Jitter folder and still more in his By the second page, you’re already building your own custom, glitchy visual filters. Great stuff.

Now, of course, this tutorial isn’t limited to users of Max/MSP/Jitter, but Max is really an ideal environment for testing shaders. (It can even work well as a prototype environment before going elsewhere.) This code will work in Processing, though, and I could see using a combination of those two tools (still working on my own workflow there).

We’ll be practicing our shader chops, because there’s definitely a need for more information like this. This sentence says it all: “If you want to learn more, I highly recommend poking around the “jitter-shaders” folder and grabbing the official GLSL specification(PDF) or the GLSL Orange Book.” Good advice, and the example Jitter shaders included with the program will already do a lot of what you’d like need. But, as fair warning, the Orange Book and other official OpenGL documentation can make your head hurt, fast. (There’s a reason “… for Dummies” isn’t on the end of the title.) I still recommend picking up a copy, but there’s definitely a need for at least intermediate documentation. Being something of a dummy myself, I may be able to help.

TyphoonLabs Shader Designer is a free IDE for creating your own vertex and pixel/fragment shaders, the magical code snippets that process 3D geometries, textures, and even images and video on your GPU. I’ve been fiddling around with it a bit on Windows as I work on learning OpenGL’s shading language, GLSL, and it seems quite handy. Previously Windows-only, it’s now available on Linux, as well. (It was developed with .NET, which is getting increasingly nice OpenGL tools of its own.)

TyphoonLabs Shader Designer

With code completion and syntax highlighting, easy access to settings, and real-time previews, this could be just the tool for someone learning about shaders for use in other software, like Max/MSP/Jitter.

If you’re on the Mac and want something similar, make sure you’ve installed the developer tools from the OS X 10.4 disc, and check out the 3D folders. There’s a similar tool for working with shaders included in the OS X distribution.

We hope to have more shader coverage through 2006, so hold onto your seats for some newbie-friendly shader tutorials. (I.e., if we can do it, you can do it.)

Via OpenGL.org’s news blog, an excellent source for all the latest OpenGL geekery.