Speaking of Adobe’s new Pixel Bender, VKNT writes about work he’s doing with polar views, as modified in an application that processes panoramic images called Pano2VR. It’s not a real-time process, in this case (though the finished result is built in Processing), but I don’t see why you couldn’t do some filter math in something like Pixel Bender or a conventional OpenGL filter that would do the same live. Could be fun stuff. VKNT writes:

This is part of my research on movement in space and new ways to represent this. It is essentially a series of 360° panoramas following a path through a space. These spherical images are converted to an angular projection, which introduces a typical distortion. The further you get away from the center, the more distortion. The center of the image is determined by 3 parameters: pan, tilt en roll. I animated these parameters resulting in a sense of movement and deformation of the space.
More info and downloadable .mov file at:
vknt.be/2008/09/25/25d-happy-new-ears-rez-08/

Ideas, dear readers?

Dan Shiffman’s Run Lola Run creation, built in Processing, appears here at Tyneside Cinema in the UK. This is how to watch a movie: 1400 frames on screen at a time. Cuts cascade across the screen, colors shift as the in-frame palette changes, and Lola’s action turns into sculptural wallpaper.

There’s no better time to mention that Dan’s Most Pixels Ever library is getting some fall semester refreshes, with new documentation and other improvements. If you haven’t used it before, this is your ticket to working on Processing with multiple displays – even if it’s just, you know, like two displays.

http://code.google.com/p/mostpixelsever/

Oh, and incidentally, it is Shiffman, not Shiftman.

Looks like Tyneside made a nice launch event for this Filament event. I love the shadows walking in the windows and colored lights. I’ve been talking with visualists lately about improving the quality of live events. Really turning something into an event, and making a visual splash on the outside of the venue, is a great way to start.

Filament @ Tyneside Cinema Launch from Steve Holmes on Vimeo.

Glenn Marshall is racking up fans on Vimeo with a new music video. He’s using Processing (site | CDM tag), but he’s built his own animation system and added lovely additive blending (that’s “nebulous” blending, as he puts it). Of course, this is what Processing excels at: you can assemble your own toolkit of creative tools, even without a lot of coding background. I bristle every time I hear people argue that tools like Adobe After Effects are “easy” while Processing is “hard”; easy might be a stretch for either one, but I think it can be a lot easier to build just what you need — and a massive, heavyweight motion graphics tool requires a lot of learning, too, in the reverse direction (top down instead of bottom up).

For his part, Glenn notes that:

• it took about 4 hours to render at HD Ed.: Okay, damn — guess real-time is out of the question here, huh?
• was made entirely using processing.org
• i could have tweaked it manually to make it more intersting, but i like the fact it’s 100% generative
• i can just about understand my own source code, let alone others, i’m still a noob programmer believe it or not.

So there you have it — this is the creation of a beginning programmer. And it looks absolutely gorgeous, I think probably because he focused on his artistic skills and design sense and wasn’t afraid to bring in some assets. He writes on his blog:

‘Butterfly’, my first film and the inspiration behind my whole direction as an artist (check out previous blog), is again the inspiration here. When making Butterfly I became obsessed with the wing patterns of the Monarch butterfly and how they looked like imaginary worlds within themselves, where butterflies lived and died according to a holistic, natural mechanism of nature.

These kind of ambitious concepts where difficult for me to implement back then within the practical limits of traditional 3d/2d software. So I wanted Metamorphosis be symbolic of my passing over into 100% programmed/generative computer art, where perhaps these kinds of ideas can reach more of their potential.

Metamorphosis [butterfly.ie]

Click through video to Vimeo for HD-quality playback. Music by Boards of Canada … yeah, my one gripe is, if you’re doing work this beautiful, go find a musician to collaborate with. I think you’ll get plenty of volunteers! And there’s even Creative Commons-licensed music to discover, as well. Entirely too much ripping off going on, and it doesn’t have to be that way. (Hey, I can’t run the site Create Digital Music and not say something!)

Previous work, same artist: Music is Math

• Windows doesn’t support capture without the addition of the buggy WinVDIG, which often doesn’t work properly.
• Linux doesn’t support anything.
• Capture is slow on Mac, and sometimes doesn’t work.
• QuickTime updates regularly hose the whole setup.
• Important QuickTime features aren’t supported.
• Playback is slow.
• Playback often crashes.
• Things just don’t work, and you don’t know why.

Now, granted, for simple sketches and experimentation, the library often will work. Or at least, it works except when it doesn’t — and you can read about how often it doesn’t in the reference.

Fortunately, I think the situation could be — and soon will be — very, very different.

The Reason Why

Here’s the important part many people miss: none of this is Processing’s fault, and all of it is theoretically fixable. Processing is based on Java, and for video support it uses QuickTime Java. Generally, video libraries in Java aren’t actually even Java; they’re just native libraries that have been supported via Java APIs. So, if it’s not Processing and it’s not Java, then it’s — you got it — QuickTime, or specifically QuickTime Java (qtjava). QuickTime Java has been dead for some time and while Apple’s rule seems to be that anything you might want to know is covered by an NDA, suffice to said it’s been recognized to be officially dead as of this year. It’s been decaying for a couple of years, but we finally got a (confidential) death certificate. Sun, for its part, never supported QuickTime Java, supporting instead something called Java Media Framework which it never finished, but which had the power to boldly not work on multiple platforms. Sun and Apple were never interested in a solution, Microsoft doesn’t really need you to use Java, and Linux people until recently ignored Java because it was proprietary. Result: a mess that left the Processing team using the one and only possible evil.

Thankfully, it doesn’t have to be that way. Open source video libraries haven’t gotten mainstream recognition because of gray areas in patent law, but technically, they’ve been leaping forward. (In fact, forget about Java for a moment — as you may already know, open source tools like VLC and ffmpeg are often more reliable and perform better on both Mac and Windows than native libraries from Apple and Microsoft. Viva le open source!)

In theory, we should also see improved video support from Sun’s JavaFX API. I’ll believe it when I see it, though, and the current status is that it can do playback on Windows only when it’s not itself crashing — and it appears to rely on external open source codec support anyway even to do that. We’re also waiting on it being exposed as Java classes, so until that happens, forget I mentioned it.

Testing GSVideo

You want something now. You want it to work. You want it to play back and capture reliably. You want high performance. And you want it on all operating systems. This is perfectly reasonable to want, because we have wildly powerful computers that are certainly capable.

Andrés Colubri’s library GSVideo seems to be the best answer for that moment. It’s designed to act just like the internal video library in Processing, so the syntax and operation is absurdly easy. But it relies on libraries that actually work. And best of all, you can couple it with his GLGraphics library to keep processing on the GPU for higher performance. (There are even HD video playback examples in the latter that connect to the GSVideo library.)

So, now you have an introduction to why I’m so excited to have this library to test. I’ve started playing with it now, and intend to test on Mac OS X, XP, Vista, and Linux by the time I’ve done. Building on Windows looks like a bit of a pain, but Windows does have very solid capture support native to the OS, so I think it’ll all be worth it. Linux looks easy, Mac OS X looks fine. I have done enough testing of GLGraphics on a MacBook to know that you’ll want a recent dedicated video card for OpenGL texture support, but then, regular readers have already discovered the non-Pro MacBook to be fairly useless for visual work.

Will You Join in My Crusade?

Here’s my question: who’s with me? Anyone else want to try this out? There’s not much action on the Sourceforge forums and whatnot, so I actually wonder if we’d like to start a small discussion and testing group and miniblog here on CDM?

If you’re interested, say so in comments. Leave your email — it won’t be published, but we can see it privately and keep you posted.

This weekend, three things that are supposed to be dead in New York — Midtown, Coney Island, and electronic music — converge in a massive festival that proves otherwise, as seen on Create Digital Music. For the night “innovation stage,” 360 degrees of generative visuals and visualist installation complement the impressive music lineup. And that brings us to some of our favorite visualists:

• Marius Watz, the talented visual programmer who just moved here to NYC
• Daito Manabe, doing light sequencing
• Paul Prudence of London (not familiar with his work, but looking forward to it) — oops, didn’t make the connection: Paul is the creator of the wonderful dataisnature.com site; I really look forward to seeing his work in person!
• Burak Arikan, a gifted and prolific Processing programmer and visual artist
• Video installations from the video art team of CTRL (Devan Simunovich and Nika Offenbac)
• Albert Hwang, working with his magical 3D light projections

It’s all good stuff — here are some samples from Burak (working with Richie Hawtin in a collaboration with Ali Demirel):

– and those light installations from Albert. They’re hard to make out in this video; maybe we can shoot some better videos in person.

Be sure to check out Marius Watz’s project Neon Organic, as well, not to mention his fantastic Generator.x site.

Innovation Night Projects
Minitek Site; Festival Tickets and Info

If you can’t make it, no worries — we’ve got a video crew covering the event for CDM, and I hope to sit down for quality time with a number of these artists soon. I’ve got to get all my New York colleagues together to help us organize the Visualist Revolution.

Mrmr is a wonderful tool for turning your Apple mobile device into a multi-touch controller for performance and controlling visuals and music. It allows custom control layouts, it’s beautifully geared to things you can do with your fingertips, and it’s fully open source. As is often the case on this site, we have two messages. One is about a specific technology to play with, the other about the broader possibilities of digital work.

The specific: Our friend Eric Redlinger has ported his Mrmr open-source OpenSoundControl multi-touch controller app to the 2.x firmware for iPhone and iPod touch. We’ve got screenshots, as seen above or via our Flickr stream, and Eric is looking for beta testers from the CDM community. (More on that in a second.)

The deeper issue: Beyond just Apple’s device, there’s a new opportunity to make controllers standard, open, and self-configuring. Why would you want to do that? Eric explains the vision:

Controlling your multimedia performance or installation with a handheld touchscreen device is cool, but what do you do when your friends want to spontaneously participate using their devices? Typically a long tutorial follows in which you explain what OSC and MIDI are and how they need to find and install a special app, then configure the server and port settings, etc. And, oh yeah, you’ll need their device’s IP number…etc.

Now imagine that conversation being like this: Go to the appstore on your phone right now and download this app. Launch the app. Play.

That’s Mrmr (pronounced murmur), and it exists already for the Mac and for the iPhone/iPod, with clients for other devices to come. Although it is not yet on the appstore, you can beta test it today. Ed.: Damn. I still want to pronounce it “mister mister.” -PK

Mrmr consists of a couple of protocols to specify the type and screen location of interface control ‘widgets’, and specifies a way to send the resulting key presses, slider values etc. back to the VJ/DJ app of your choice. It uses standard OSC for its messaging protocol so it works with any existing app that supports Open Sound Control support.

What this means for you is that you can design a custom interface for your Max/MSP/Jitter / Pure Data / Quartz Composer / etc. environment and push that interface onto your phone, and onto others’ phones, providing a great new way to add multi-user, collaborative elements to your set!

And, of course, this ultimately has implications not just for the multitouch Apple mobiles but future multitouch technology, too.

Project page / wiki: http://poly.share.dj/projects/#mrmr

How to get involved in the beta: Eric is definitely looking for testers. now has the testers he needs! Stay tuned!

You’ll need to email your device ID of your tester iPod touch or iPhone running the 2.x firmware. There are two ways to go about that. Here’s a set of instructions for how to find the ID:

Providing Your iPhone Device ID to a Developer

If you use that approach, be sure to put “mrmr beta” in the subject header.

Even better, Erica Sadun has built an app for the job.

Ad Hoc Helper [iTunes Link]

Download it, run it, and it automatically sends off an email with the ID with the subject line already filled in.

Either way, address your emails to eric (at) share [dot] dj with the ID — and let us know how it goes. We hope to have more support materials up on using mrmr very soon, so stay tuned.

Updated: Eric’s testing list is full! But while the beta testers and Eric work on making the app stable in preparation for release, do stay tuned — we’ll have quite a lot more on OSC and how to use it soon, and will keep you posted on official mrmr for firmware 2.0 availability!

If you still want mrmr right now, it’s available on jailbroken 2.0 firmware via Cydia.

I believe that coding is an essential skill for people making live digital visuals. At the same time, there’s no question that learning to code has been a big obstacle for visually creative people — especially as they have plenty of other things on their mind. You need somewhere to start, and you need to make the learning curve manageable. Processing has been a great tool for doing that, but the point isn’t to learn Processing — it’s to learn how to code.

For that reason, I’m thrilled that Dan Shiffman’s book Learning Processing is now in print and available. I’ve been waiting for this for some time. There are already a couple of great books out there for Processing, but Dan’s book is unique in that it’s entirely focused on teaching you to code visuals step by step, even if you haven’t coded before. Dan teaches coding to creative-minded non-coders at NYU’s ITP program, and the book comes out of that teaching technique.

I know the book very well as I served as a technical editor during its development. We’ll be running an exclusive set of excerpts this week, but here’s a look at what’s included:

• Basics of code structure, pixels, interaction, and fundamentals like how the coordinate system works
• Using arrays to make lots of stuff appear on the screen, including particle systems
• Basics of images, video, data, and networking
• How to use object-oriented programming to make coding easier and more efficient
• Extending Processing with Java and more advanced coding techniques

By the time you’re done, you’ll be processing pixels, drawing generative visuals, and writing well-organized code.

To me, one of the real strengths of this book for teaching and learning is its strong emphasis on object-oriented programming, in a way people can actually understand. Explained properly, objects can really help keep your code clean. For some reason, this is often viewed as an “advanced” concept, but on the contrary, I’ve found using objects actually helps keep beginners from getting tripped up. And, to put it in visual terms, understanding objects is a terrific skill for getting lots of glitzy eye candy up on the screen.

Here’s what Dan has to say about the book (emphasis mine):

My goal for “Learning Processing” was to write something for the complete and total programming beginner. If you’ve never written a line of code before in your life, but want to get started creating your own digital media tools then I wrote this book for you. There are several other wonderful Processing books out there and I hope mine will complement them nicely. A special thanks to Casey, Ben, and Ira who kept encouraging and inspiring me as their books were being published.

The book is also geared towards the teacher. It’s not my belief that such a person will necessarily learn any new skills from the book (assuming they have a programming background), however, my hope is that the book will encourage and help facilitate the teaching of programming. It is structured with 10 lessons (complete with examples and exercises) and can act as a ready-made syllabus for a beginner interactive media / programming class. In fact, the book is modeled exactly on ITP’s Introduction to Computational Media course.

The book is available on Amazon.com. It shows out of stock, but those kind of screw-ups are common when a book has just come out (speaking as a published author here).

Stay tuned for those excerpts later this week; I’m editing them now!

See also: Dan Shiffman’s blog, Facebook Page, Official Site, free download of TOC and first chapter

Wiimote Headtracking in Processing via wesen’s Twitter (follow CDM at Twitter: cdmblogs)

Once the data is there, the rest is basically math. You position the camera to look straight ahead, and then adjust the viewing angle based on incoming data.

The sketch is available, so go try it — and see if further improvements or other applications are possible.

In case you aren’t already familiar with it, here’s the now-famous video featuring Wiimote headtracking, by Carnegie Mellon’s Johnny Chung Lee:

As code literacy improves and coding tools like Processing and Flash make it easier to produce stunning visual results, the line between the coder/hacker and digital artist, and more conventional artists, is blurring fast. The next trend: networks and blogs on which people share not just their work, but the code behind it. The idea is old, but there’s no question the breadth of content and number of participants is expanding - and beginners are welcome, too.

The Flash Virtuoso, and Galleries vs. Code Repositories

Keith Peters, aka BIT-101, has been instrumental in the Flash community in advocating digital art and animation. His books are clearly written and intuitive to non-programmers — despite their Flash basis, I’ve found them useful for my Processing experiments, too. And Keith has been busy of late. He’s got a second installment coming for his wonderful Making Things Move book, inspiring his isometric experiments pictured here, and he’s also launched a new site called “Art from Code.” (Various permutations of this theme come up regularly.)

I owe a huge debt to Keith, as I got into generative coding entirely through his books, before later going on to discover Processing.

Interestingly, the relationship between code and art is an imperfect one. Just open sourcing the code isn’t always practical. In a way, though, that makes the code even more beautiful — and sometimes sharing visual results can be just as interesting as sharing code. (It forces us to go back and try to reproduce the results, then get it all wrong, and wind up producing something original, often as a result of mistakes!)

Keith writes on his blog:

I want this site to be more about the images than an open source code repository. Not that I’m against sharing the code, but I don’t want that to be the focus here.

Two is that the code that has created most of the images here is ephemeral. I open up Flash or Flex Builder, start messing around, take some screen shots when something looks good, keep messing around and before long the code that resulted in the first image is long gone, having morphed into something else entirely. I may even leave it off in a completely broken state, or in some cases, if I’m working in Flash, I never even end up saving the FLA with the code in it. So 99% of the time, sharing the code that created image X is impossible.

Also, interestingly, Keith notes he often works with code directly on the timeline — something that’s not possible in Processing. (I’m personally guilty of making various unfair comparisons between Processing and Flash; the bottom line is, you might as well compare watercolor and oil paint. It doesn’t matter. They’re different. And I love that Keith works totally differently from the way I do.)

Keith does, however, promise some new graphics classes, and I’ll be very eager to translate them to the world of Processing, perhaps — again — imperfectly, resulting in something else. I’ll be watching and will let you know when there’s some code to go with the art.

The Open Processing Gallery, with Code

If “Art from Code” is intended as a gallery first, rather than a code repository, OpenProcessing is both, simultaneously. OpenProcessing is the work of Sinan Ascioglu, an NYU ITP student (I believe he has other contributors, as well). Sinan has also built OpenVisuals, a site dedicated to easily visualizing data in a consistent way, also using Processing:

OpenProcessing
OpenVisuals

OpenProcessing is a basic gallery with code for Processing sketches of all kinds. OpenVisuals goes one step further, providing a framework for data visualization — thus, it’s not only a gallery, but a platform on which you can quickly see the results of turning data into something onscreen. Because Processing sketches can be easily exported as embedded applets, you can immediately see live, real-time results on the webpage, while comparing the open-sourced code that makes it function.

With a variety of people contributing, part of the joy is seeing the range of expression, not just from experienced coders but newcomers, as well. It’s also encouraging to see there’s embed capabilities, so you could embed the uploaded results onto a blog entry or forum post.

For a similar framework for data visualization for Flash — though without the accompanying online community — see the open source Flare library. (Flash is not truly open source, but work can be done in Flash that is, of course.)

IBM Visual Communication Lab’s ManyEyes is a similar community built on working with data visualization, and makes some use of Flare. (It’s also a great data source for Processing projects and the like!) And for six degrees of Create Digital Motion, one of the team at IBM is Jesse Kriss, who built Processing-Max communications conduit MaxLink.

OpenCode, the Web Browser as IDE

Embedding applets and text code is one thing. Turning your browser into a live development environment is another. OpenCode takes the online gallery - code repository - community to its extreme conclusion, by allowing you to edit and run code live without leaving your browser. (Yes, that’s right — just in case you can’t be bothered to copy and paste into a Processing window.) With enough abstraction behind the scenes, it could eventually be made friendly to non-coders, too. Like OpenProcessing and OpenVisuals, the tool is built on the friendly, elegant, and lightweight Processing platform.

OpenCode

It’s a fascinating idea, and impressively, it works quite well. It could be just the beginning of what’s possible - imagine people wanting to work with data being able to use a couple of lines of live code, making modifications right in their browser, while libraries installed on the server abstract all the other functions behind the scenes. (And yes, Java-based libraries do work.)

Of course, cool as that is, somewhere along the line I think we actually lose some of what Keith (and others, like our visualist hero Robert Hodgin) have done. Wonderful as it is to be able to share code, it’s really the visuals that are themselves the payoff. I’d love to see visuals first, code second instead of the other way around.

OpenCode is in pre-alpha state, the work of Kyle Buza and Takashi Okamoto at the MIT Media Lab. I love their tagline - “This is so alpha, you won’t even want to use it™.” So consider it a work in progress. (I met Kyle and Takashi in Boston earlier this year; very nice, talented guys! Kyle earns extra points as the creator of chiptune externals for Max.)

Perhaps the MIT and NYU projects will meet somewhere in between. I propose a conference somewhere on the Amtrak route between the two cities — Mystic, Connecticut, perhaps?

What’s Next?

Flash and Processing are really ideal for these applications, because of their accessibility, and the fact that they can run online and produce immediate visual results. But, powerful as I think data visualization can be, I don’t think the future of this is limited to infographics. The superset is clearly art. And the networks like those above are only likely to multiply and grow.

So, artists, what would your ideal community look like? What sorts of sharing features would you want in a tool like Processing, or on the online side? Or is there a point where you actually don’t want to see / share code, preferring to focus on the actual artistic result?

Ed.: The JOGL changes are a big deal; I’m curious to see what improvements this may cause in render quality via the OpenGL renderer. Processing is nearing its real (non-beta!) 1.0 release, so expect a full-blown update on the progress of the tool soon; see Ben Fry’s recently-posted status report. -PK