Using video in Processing is, sadly, really painful. You can do absolutely wonderful things once it’s working — pixel-by-pixel manipulations that are hard to do elsewhere, and easily-coded OpenGL manipulations that should help generate powerful eye candy. But the list of issues runs something like this:

  • Windows doesn’t support capture without the addition of the buggy WinVDIG, which often doesn’t work properly.
  • Linux doesn’t support anything.
  • Capture is slow on Mac, and sometimes doesn’t work.
  • QuickTime updates regularly hose the whole setup.
  • Important QuickTime features aren’t supported.
  • Playback is slow.
  • Playback often crashes.
  • Things just don’t work, and you don’t know why.

Now, granted, for simple sketches and experimentation, the library often will work. Or at least, it works except when it doesn’t — and you can read about how often it doesn’t in the reference.

Fortunately, I think the situation could be — and soon will be — very, very different.

The Reason Why

Here’s the important part many people miss: none of this is Processing’s fault, and all of it is theoretically fixable. Processing is based on Java, and for video support it uses QuickTime Java. Generally, video libraries in Java aren’t actually even Java; they’re just native libraries that have been supported via Java APIs. So, if it’s not Processing and it’s not Java, then it’s — you got it — QuickTime, or specifically QuickTime Java (qtjava). QuickTime Java has been dead for some time and while Apple’s rule seems to be that anything you might want to know is covered by an NDA, suffice to said it’s been recognized to be officially dead as of this year. It’s been decaying for a couple of years, but we finally got a (confidential) death certificate. Sun, for its part, never supported QuickTime Java, supporting instead something called Java Media Framework which it never finished, but which had the power to boldly not work on multiple platforms. Sun and Apple were never interested in a solution, Microsoft doesn’t really need you to use Java, and Linux people until recently ignored Java because it was proprietary. Result: a mess that left the Processing team using the one and only possible evil.

Thankfully, it doesn’t have to be that way. Open source video libraries haven’t gotten mainstream recognition because of gray areas in patent law, but technically, they’ve been leaping forward. (In fact, forget about Java for a moment — as you may already know, open source tools like VLC and ffmpeg are often more reliable and perform better on both Mac and Windows than native libraries from Apple and Microsoft. Viva le open source!)

In theory, we should also see improved video support from Sun’s JavaFX API. I’ll believe it when I see it, though, and the current status is that it can do playback on Windows only when it’s not itself crashing — and it appears to rely on external open source codec support anyway even to do that. We’re also waiting on it being exposed as Java classes, so until that happens, forget I mentioned it.

Testing GSVideo

You want something now. You want it to work. You want it to play back and capture reliably. You want high performance. And you want it on all operating systems. This is perfectly reasonable to want, because we have wildly powerful computers that are certainly capable.

Andrés Colubri’s library GSVideo seems to be the best answer for that moment. It’s designed to act just like the internal video library in Processing, so the syntax and operation is absurdly easy. But it relies on libraries that actually work. And best of all, you can couple it with his GLGraphics library to keep processing on the GPU for higher performance. (There are even HD video playback examples in the latter that connect to the GSVideo library.)

So, now you have an introduction to why I’m so excited to have this library to test. I’ve started playing with it now, and intend to test on Mac OS X, XP, Vista, and Linux by the time I’ve done. Building on Windows looks like a bit of a pain, but Windows does have very solid capture support native to the OS, so I think it’ll all be worth it. Linux looks easy, Mac OS X looks fine. I have done enough testing of GLGraphics on a MacBook to know that you’ll want a recent dedicated video card for OpenGL texture support, but then, regular readers have already discovered the non-Pro MacBook to be fairly useless for visual work.

Will You Join in My Crusade?

Here’s my question: who’s with me? Anyone else want to try this out? There’s not much action on the Sourceforge forums and whatnot, so I actually wonder if we’d like to start a small discussion and testing group and miniblog here on CDM?

If you’re interested, say so in comments. Leave your email — it won’t be published, but we can see it privately and keep you posted.

  • Daniel

    Peter. Your site is invaluable.

    This Library works and works well on Linux. As you noted, it's the only way to get video in Processing on a Linux box. I've tried it for Pd and I plan on moving on to more intense trials in Processing with it. Hopefully everyone will eventually ditch the platform dependent stuff and this library becomes more user friendly for people to install.

    Have fun. Looking forward to seeing the results of your work.

    Daniel

  • http://createdigitalmusic.com Peter Kirn

    Thanks, Daniel.

    The platform specifics I think are necessary; with capture, for instance, it's your only route. But then again, if someone just builds the binaries regularly and packages it up, that'll make it easier. I haven't looked into licensing there, though, as that'd be a major factor … maybe someone else knows. I'll keep everyone posted, but figured I'd go ahead and launch the discussion. :)

  • http://studioartist.tumblr.com john dalton

    I'd be interested in keeping posted on what is happening with this, so please add me to your list of interested parties.

  • http://abstrakt.vade.info vade

    Myself as well. One of the reasons I initially ditched using processing is its lack of realtime video manipulation. Id love to see a sample app that demos some of this functionality, only because im curious how it stacks up performance wise to some of the other solutions. Id love to get my hands dirty with all of the other libraries processing currently supports, as Quartz Composer doesnt have nearly the caché of interesting thing Processing has. Someone needs to port OpenFrameworks and some P5 code to Cocoa/QC, so I can use it. Anyone up for that ;)

    Nice writeup.

  • http://createdigitalmusic.com Peter Kirn

    Well, I think to the extent that you're doing stuff on the graphics card, you should get roughly the performance you'd get with anything else … though part of the *advantage* of these two libraries is that then you can go pack to a PImage object, and at that point I'd expect a performance tax again.

    As for connecting this back to Cocoa… no idea there. I mean, you can use C++ and (minus the handy Cocoa bridge) even Java libraries within a Cocoa project, can't you? It's not going to be "pure Cocoa," if that's a concern. But both OFW and P5 use OpenGL pipelines.

  • http://abstrakt.vade.info vade

    Well, not all texture upload and video decompression libraries are created equally Peter. Just because its doing stuff on the graphics card does not mean performance will be identical. ;)

  • http://createdigitalmusic.com Peter Kirn

    Well, of course — I'm just saying, video performance is going to be similar to GStreamer, and the stuff it is doing in straight GLSL is pretty vanilla.

  • http://abstrakt.vade.info vade

    Well, I was asking in relation to other apps that, well, a VJ might use ;) . Thus a simple, pre-packaged example app to test some theoretical performance.

  • juan

    All this sounds really good. I'd really like to know how this is developing. Please add me to your list.

  • http://xsvdo.vox.com/ xsvdo

    I'm in! I was moving from Processing to OpenFrameworks because of lack of video support. However, this looks promising. Please keep me posted (downloading via macports now).

  • naus3a

    i'm totally in this thing!!! i love doing visuals in processing and i hate the fact that, now that i finally have a webcam which likes my linux box, i can't sample and augment video input.

  • http://myernore.com tacitdynamite

    I'm definitely interested. I've been using jmyron library, and it's okay, but would be thrilled to see something better. It looks like the GSMovieMaker class doesn't work yet? that's a shame … how can we use it if this isn't working yet?

  • http://createdigitalmusic.com Peter Kirn

    GSMovieMaker is the *output* class; that is, the equivalent of the screen capture library Dan wrote for output of non real-time video from your Processing sketch. Video input and video playback are both working which are really the important bits and the ones that have to be real-time. So GSMovieMaker will be icing once it's done. ;)

    JMyron is nice but not so actively developed, and it doesn't do playback, it does native capture only on Windows, and really is best suited for analysis. So this is huge having GSVideo.

  • Marcus

    If GSvideo is like GStreamer it will work and it will work freakishly well and efficiently and it's easy to program for!

  • etienneg

    I'm in too !!! way less expert than you all, but with strong will to work on tv-related interface prototyping !!

  • Damon Baker

    I'm in. I've been exploring using gsvideo/p5 for a virtual reality project i'm working on

  • http://experimentaclub.com/druhb Jaime Munarriz

    I stopped using P5 for live video becaude QT was eating as much as 96% of my CPU. I switched to Isadora, tried VVVV, then Jitter, and I'm currently using Pd/GEM.
    But there're so many things you can do with P5 that you cannot do with patch programming, that I'm really waiting for a good video library. So please, we need it!!

  • http://www.squarenoise.net Rien Daamen

    I’d be interested in keeping posted!
    Having started and stopped working in P5 several times and looking for alternatives. To me, a proper fps is essential.

  • http://mxmai.com mxxx

    i'm working on a project at the moment in processing which involves recording one video whilst playing back another, and it's slowing to a crawl. 15fps at 640×480 on a macbook pro.

    i'm gonna give gsvideo a run this weekend and see what i can get out of it. i'll report back on monday.

  • http://codeanticode.wordpress.com/ Andres

    Hi guys,

    I'm Andres Colubri, the creator of GSVideo. It's great to see people interested in the project, so please post your experience with the library, bugs, comments, suggestions, etc.

    Since GStreamer in natively developed in Linux, is in this platform is where you might have the most seamless experience, specially in terms of installation (GStreamer usually comes pre-installed by default in all major distros). Peple have been able to do video capture using either v4l or v4l2.

    On Windows and OSX the story is a little bit more complex, in terms of having GStreamer properly installed on the system. I have packaged myself an up-to-date, simplified GStreamer windows installer which is available here, along the latest "preview" version of gsvideo.

    And for OSX, the only way so far of getting GStreamer is to use macports. It would be nice to see a GStreamer package installer for Mac…

    Andres

  • http://mxmai.com mxxx

    installing GStreamer via MacPorts is pretty simple (there's some good instructions here.

    unfortunately the project i'm working on at the moment takes a massive performance hit using the MovieMaker class, and your GSMovieMaker class is unavailable at the moment. but in terms of straight up movie playback, the GSMovie class was definitely faster than the built in processing libraries. i did have problems utilizing the OpenGL rendering via GLGraphics, though.

  • http://codeanticode.wordpress.com/ Andres

    I had to put the GSMovieMaker class on hold until I finish some other work… Even though macports makes possible to install gstreamer on OSX, it would be better to have a installation package which doesn't require compiling from source. Right now, creating such installer is my priority for the Mac version of gsvideo.

    What is the problem you are encountering with GLGraphics?

  • Pingback: Scrambled video test and hopes for a stable video library « NYX EDITION

  • http://californiarealestatelicensesearch.blogspot.com california real esta

    Some interesting blog…

  • kocosman

    great topic. i’m highly intersted. my mail is: kocosman@gmail.com

  • sebas

    Someone can send me the librery GSVideo? I can not download it from the website

  • deqmute

    can you show me how to install GSvideo on Processing? Something go wrong. It cannot function properly