Jitter works brilliantly when it comes to processing signal - and that means for signal-like work with video and textures, it’s fantastic, as well as the usual Max-y tasks like processing input from physical sensors and input devices and the like. But try to do a whole lot of sophisticated 3D work, and Jitter may not be the best tool. For game-style 3D graphics and interaction, you want some standardized rendering and scene graph tools to take care of the hard work, plus physics and other capabilities that bring together your 3D scene.

That’s why [myu], the Max - Unity Interoperability Toolkit, looks so appealing. It not only allows for bi-directional data integration (via TCP) of Max and the Unity game engine, but can dynamically pass textures between the two. For those of you comfortable patching, say, chains of shader processors in Jitter, that means you can very quickly add some of the tasty 3D scene powers of Unity. Put together your textures in Jitter, and, say, dynamically process input from a Wii Fit balance board, then bring the input data and textures into Unity. (Unity is a friendly, elegant game engine built in C# and Novell’s open-source Mono implementation of Microsoft’s .net. Unity had previously been Mac-only but with a major new release now runs on Mac and Windows.)

The toolkit is the result of research at Virginia Tech Interactive Sound & Intermedia Studio director Dr. Ivica Ico Bukvic.

Needless to say, this could have powerful implications for all kinds of live and interactive installation applications. And yes, it is all released under the GPL.

[myu] Max-Unity Interoperability Toolkit v.1.0 Released [Cycling '74 Forum]

More Max+Unity Game Engine Goodness, with Powerful Toolkit for Max, Jitter, Pd

Teaching Adaptive Music with Games: Unity + Max/MSP, Meet Space Invaders!

For other examples of combining Max and Unity - in this case for Max’s musical powers and Unity’s gaming prowess - see another story from today:

Teaching Adaptive Music with Games: Unity + Max/MSP, Meet Space Invaders! [Create Digital Music]

Updated: About those textures…

Ico follows up to answer our questions about how you might use textures with Jitter and the Unity Game Engine, via his [myu] toolkit:

videoprojectiontools, the powerful and intuitive Max/MSP/Jitter-developed Mac and Windows tools for projection mapping, just got a nice update. The new version has OpenSoundControl (OSC) support - and yes, despite the “Sound” in the acronym, it’s really more like Open … Control. Max is not required to use the patches; they’re standalone.

Yet again we see some advantages of using OSC:

The implementation so far for OSC includes preset and cuelist access, and layer fades,pos x and y, scale x and y, and videotracks selection from the individual sources.
With OSC you can now sync several computers and trigger presets from a OSC-able application (which can run as a background application).

Head to the site for downloads, tutorials, and documentation to get you started.

Video projection tools

Closing out our (unexpected) Week of Optical Flow feature, Michel from the vvvv forums has ported Andrew Benson’s optical flow implementation to that DirectX-based, Windows-only, free for non-commercial-use patching environment. (Jitter and Quartz Composer, mentioned earlier, each use OpenGL, not DirectX.)

Since it is DirectX, the shader uses HLSL instead of GLSL. In technical terms, folks, that means it’s one letter more different. I can illustrate: Gary the gregarious gorilla would become Harry the hreharious horilla. You see? (Kidding. HLSL is actually closer to NVIDIA’s Cg, but all these things are built on the same basic principles - and they’re all just shader languages.)

This is just a first go at this patch, so vvvv users wanting to improve upon it, go for it!

Distortion From Optical Flow incl. Optical Flow Shader

Add Max patch effects to DNA. from Livid Instruments on Vimeo.

Yesterday, we saw some splashy video distortion techniques applied to real-time video. You know what that means: it’s time to use these in live performance.

Liquidify Video, Live: Optical Flow GLSL Datamosh Technique

Here’s one start.Peter Nyboer, Max whiz and Livid developer, has run with the idea of squishing around video using optical flow analysis, and shows you how to add the effect to Livid’s Cell DNA VJ app. For Jitter users, this means you can rely on Cell for quick access to video taps and files, while adding unusual effects built in Jitter to get your custom processing on, not only with this example but any other patches you’ve created. One little detail of Cell DNA I missed – it requires Max 4 patches, not Max 5 patches. Peter has also posted a tutorial for working with that, after the jump.

And yes, if none of this is really making sense to you, you can go download the files and just try it out – no need to fully grasp all of the internals straight away.

Don’t want to use Jitter and/or Cell? The guts of Andrew Benson’s video datasplooshing technique is an OpenGL (GLSL) shader, so it doesn’t even rely on Jitter – Jitter can just be a convenient environment for playing around with such things. There’s word we may see a Quartz Composer wrapper around this shader, which would make it easy to use with software like VDMX.

Oh, by the way, I’m officially rescinding my editorial ban on the term “datamoshing.” Why? Because it means absolutely nothing, and therefore can be declared reasonably harmless. Also, unlike the term “glitch,” it comes without any baggage. We therefore have a nice, nonsense term for making video all mushy and unpredictable – a good thing.

motion distortion 2 from andrew benson on Vimeo.

Datamosh? (The “forbidden” but harmlessly meaningless word?) Video squishification? Mushy data?

Call it what you will, but applying real-time distortion and displacement to video so that video textures become flowing layers of pixels looks absolutely beautiful. Andrew Benson of Cycling ‘74 has only just begun playing with this in Jitter using GLSL shaders, and already the results are really compelling. (For a simpler example that looks more like the compression artifact technique we’ve seen recently, have a look at the second video – though, personally, I like the more sophisticated, layered approach of the video at top. This is going some very cool places.)

This is a Jitter patch, but would be simple enough to port to code for Processing, FreeFrameGL (which implements shader code), or other tools, too, in case you can’t bear being away from your moshness.

Andrew writes: