I believe that when we hear music, we employ visual centers in our brain in the way in which we experience its sensations and understand its patterns. But actually visualizing music and sound on the computer is a wonderfully impossible project, fraught with challenge. Somewhere in the struggle to do so, terrific things can happen. And the work of the Abstract Birds collective and Quayola is some of the nicest work I’ve seen yet.

The thing with feathers: Partita variations, demonstrating the flexibility of the system to take on differing visual looks – essential to its use as a performance tool and with different music.

I was fortunate this weekend to spend some time this week at the Elektra Music festival in Montreal with the Abstract Birds crew. I’ll have a full write-up/review of their Les Objets Impossibles, as well as some ideas about visual music floating around the event, but I want first to look at Partitura 001, which works in real-time responding to music. Built with some serious 3D visual chops in the graphical patching environment for Windows, vvvv, Partitura is a virtuosic audiovisual creation, fit perfectly here to the music of Telefon Tel Aviv.

The biggest question in translating music to space is determining how to measure the dimension of time. (Arguably, this is one of the great questions for humanity in the past 150 year.) Partitura employs the simplest, but perhaps the most elegant, solution. Like a score, time is an arrow, a one-dimensional continuum on a single axis. Here, like intertwined DNA, musical events flow across the screen. Printed scores wrap from top to bottom, mapped to the traditional codex, but they, too, are realized in time by moving along a path in space, a forward arrow. And we have similar mechanical solutions: the cassette tape and reel to reel advancing forward like the visual here, or, wrapped into a spiral, the forward line of a phonograph.

The significance of Partitura, though, is really in its variation, its ability for customization. Here, synesthetic (and synthetic) possibility can be realized in choices of color and form. A few parameters can create vastly different aesthetics. Speaking to people watching Abstract Birds’ work in Montreal, I found that folks who weren’t necessarily musical specialists or even terribly familiar with audiovisual performance and visualization were deeply concerned with those sorts of questions. Why would a certain form or color go with a certain sound? Any system that does visualization would do well to consider engineering in these sorts of parameters for control, and experimenting with different combinations.

The linear flow of time from right to left or left to right is familiar from mechanical and analog devices, as with the Steenbeck film editor. In fact, since you don’t visually see the movement through the space of a written score, you could imagine that film here is the predominant inspiration for work like Partitura. (Side note: boy, this photo makes me remember screwing up film threading a couple of times. Ahem.) Photo (CC-BY) Marcin Wichary.

Here’s what the creators say in their description:

Partitura is a custom software to generate realtime graphics aimed at visualising sound. The term “Partitura” (score) implies a connection with music, and this metaphor is the main focus of the project. Partitura aims to create a new system for translating sound into visual forms. Inspired by the studies of artists such as Kandinsky, Paul Klee, Oscar Fischinger and Norman McLaren, the images generated by Partitura are based on a precise and coherent system of relationships between various types of geometries. The main characteristic of this system is its horizontal linear structure, like that of a musical score. It is along this linear environment that the different classes of abstract elements are created and evolve over time according to the sound. Partitura creates endless ever-evolving abstract landscapes that can respond to musical structures, audio analysis and manual gestural inputs. It is an instrument that visualises sound with both the freedom of spontaneous personal interpretation/improvisation and at the same time maintaining the automations and triggers of mathematical precision.

Partitura defines a coherent language of its own for the creation of new contemporary abstractions. It is within this system that Partitura creates worlds that expand from a single dot to multiple galaxies, from minimalism to complexity, from rigid to elastic, from solid to liquid, from angular to smoothness, from tentative to boldness, from calm to agitation, from slow to fast, from desaturated to saturation, from dark to lightness, from predictable to unpredictability. Literally ‘everything’ and its opposite… just like a musical flow.

Pedro Mari and Natan Sinigaglia are Abstract Birds, working with London-based Quayola.

It’s worth seeing some of the earlier iterations of Partitura on Quayola’s site. Iteration here is really significant, as – like layers of paint producing photorealistic effect – layers of shaders and successive enhancement of the visual idiom have made the work both more beautiful and more expressive of the music. And intriguingly, Quayola early on worked with vertical, not horizontal lines, in a series called PTA. Of course, amidst the abandoned experiments of one artist you might find the beginnings of a different path in your own work.

There are many ways to track the history of visual music, but one interesting thread, so to speak, would be to trace the use of these sorts of temporal approaches, the one-dimensional flow along a line.

http://www.abstractbirds.com/partitura/
http://vvvv.org/

  • mitch

    I can see how the below videos are real-time but are you positive that the top video is real-time as well? Have we developed that far into real-time shading?

  • http://danwinckler.com Dan Winckler

    @mitch: yes, this kind of realtime rendering is very much doable at the present time. In fact, not to knock the video above (which I really like) but you can see much more complex realtime rendering in today's demos, e.g., at pouet.net. Here's one behind-the-scenes that's particularly illuminating: http://directtovideo.wordpress.com/2009/09/30/fra… — and that's two years old!

  • http://buggyboy.onorbumbum.com buggy boy

    I wish there was a way to test this software! I WOULD LOVE TO USE IT LIVE!

  • Peter Kirn

    @buggy: Believe it. It's actually not so much how we've developed in hardware as the fact that these folks have some serious chops. ;) I talked to them a bit about their shader pipeline. It definitely helps having more capable cards, but the fundamental techniques aren't new – it's more about someone knowing how to apply them artistically for the best results, and it's a *lot* of successive shader tricks that make it look good / make the lighting look realistic. The rest really has to do with what our eyes find pleasing.

  • Peter Kirn

    Oh, and PS, based on what I saw at Usine at MUTEK, this stuff really deserves to be seen in full rendered HD, not just what you get on Vimeo. Looked utterly brilliant on a giant screen.

  • http://buggyboy.onorbumbum.com buggy boy

    @peter I'm not only impressed from the shading I also loved how they implemented the sound analyzing algorithm. When I first saw the video I thought they used MIDI.

  • http://video4vj.com VJ Tatoomi

    Beautiful! Thx to share

  • dave

    the possibilitys of todays personal computer hardware is amazing. and this shader stuff is not too different from gerneral artistic tools like max or puredata. not harder to understand or something, just one additional building block. i think there is a lot of potenzial left for doing gpu based art.

    vvvv integrates the graphic card very well and let you do all kind of crazy things. abstract birds nice work speaks for it.

  • http://dietervandoren.net dtr

    amazing. watching that video made me happy. (and i too thought it was midi triggered)