Keith reaches into Quartz Composer's magic

When they say ‘he lives in the computer’..

Quartz Composer, first introduced in OS X 10.4 and rooted in the underground visual app PixelShox, is a hidden gem in Mac OS X. This free visual tool makes some sophisticated video and 3D magic possible without coding. While promising, the version in 10.4 had some significant shortcomings. We turn to Keith Lang of developer plasq, creators of Skitch and ComicLife, to give us an inside look. Keith is not a programmer — so he’s all the more eager to go the visual patching route — but he has had his hands on QC for some time in Leopard dev builds. And now, he’s free to tell us all he knows. -PK

Mac OS X 10.5 “Leopard” is here, and the initial hoopla about the reflective dock has died down. Now that we’ve all passed through the Seven Stages of Acceptance, we have an opportunity to take this brief slither of quiet to delve into the subtler changes within the big new cat. Those changes include improvements made to Quartz Composer, the quaint and oft-misunderstood visual processing environment bundled with the Developer Tools.

What You See: The Interface

For the uninitiated, Quartz Composer is an ‘environment’ tool which lets you patch together various elements in order to build screensavers, performance environments, and animated windows for use within an application. If you’ve ever looked at an Apple screensaver, played with the new iChat Effects or watched photos float and twirl on an Apple TV, then you’re familiar with what Quartz Composer can do. Its peers are tools like Pd (Pure Data), Processing, and Max/MSP/Jitter, with Quartz Composer heavily geared towards motion animation of processed video and images. Input can be everything ranging from your iSight to XML to audio to a Wiimote, with output being a window on 3d space containing all the sprites, effects and movement you care for. Because Quartz Composer is a real-time system, you can plug stuff together whilst watching and tweaking. The underlying engine is very efficient – and with a little work you can create teh cool, without ever needing to type teh code.

Quartz Composer interface

A new look interface partially reduces multi-floating window hair-tearing.

What’s New in 10.5

This article assumes you’re familiar with Quartz Composer in Tiger, if not, you can read up on what other people are doing with it. The first thing you’ll notice is the updated user interface. Finding and placing patches is now less work in Leopard. Search for your patch, press Return, and presto! — it appears in your workspace. The patch parameter inspector can now be displayed inside the main windows, but QC persists with its annoying floating inspector and patch creator windows. Along with the Viewer window, these are prone to getting lost under other applications and documents. Ultimately, I’d love Quartz Composer to model the new iMovie layout, with an option to detach the viewer window… but Ways to Improve the QC Interface to Reduce Hair-Loss is entirely another article.

When you launch Quartz Composer, you are presented with an improved template system, which should help both the new and experienced. And of course, the biggest new help to a user is…. Undo. Yes, crazy talk, I know. But you can now Undo when working within Quartz Composer. QC now lets you Zoom in and out freely on the interface as you open multiple Settings windows, keeping your eyes on non-disappearing patch inspector whilst copying variables with wild abandon. And, you can now document your experience with colored notes under your patches to describe what precisely the spaghetti is supposed to do. Speaking of which, patch leads are now easier to manage, with a click-to-start, click-to-end action. Also for your patching convenience: a menu option to select ‘Upstream’ and ‘Downstream’ patches and a Compare Compositions mode.

Some of the smallest fixes will be the most appreciated. The Inspector window used to forget its selected pane, requiring you to continue to select the pane you needed … every … single … time … you … selected … a … new … patch. This is now fixed. Huzzah! You can also now adjust your preview window to a specific ratio, such as 4:3, to make sure your composition will work in its final space.

Changes Under the Hood

Aesthetically, Quartz Composer on Leopard is similar to Quartz Composer on Tiger. The interface is a little better, but much like Leopard itself, it’s the changes to the underlying technology which excites. According to Quartz Composer’s Release Notes, “the Quartz Composer image pipeline has been completely redesigned…”. Here’s what that means:

  • Big ass images! Images used in Quartz Compositions no longer need to be smaller than VRAM – it’s all handled dynamically for you. This is pretty damn cool, as it allows you to feed and manipulate hi-res images with abandon
  • Custom-coded patches: You can now officially write your own patches using Objective-C and the QCPlugIn / QCPlugInViewController APIs. Look at the large number of Widgets available from 3rd parties for a sense of the impact this will have on the patch lexicon. Ed.: I agree — and the thing has only just come out; I expect there will be more. Note that what QC calls “patches” are what every other app calls “objects”, so that’s confusing. -PK
  • EZ-Textures: Texture mapping and related functions are much improved and simpler to use
  • In living color: Accurate Color management for all images, for the first time
  • Better performance: General CPU usage optimizations, which is always good
  • QC, dev tool: For the programmers out there, Quartz Compositions are now even easier to integrate with your code

Fortunately, Apple uses Quartz Composer for its own projects more and more, which encourages them to make their “dog food” taste better. As a result, almost all the patches have been improved and tweaked in some way to make doing the most common things easier.

Structure

  • Compose with compositions, perfect for performance: It’s now possible to use and load compositions within compositions. This lets you more easily piece together complex work, and use a single composition to VJ its way through others.
  • Feedback: You can now connect patches in circles – A to B to C to A. This means you can build all kinds of systems, including feedback loops with ANY kind of data.
  • Sample and Hold, Watcher, Queue and Smooth. These new patches makes controlling the data flow in your composition much simpler – oh, the number of times I’ve wished I had a way to smooth data!
  • Timelines: I made all the object and camera motion in the Skitch demo movie introduction using Quartz Composer’s Interpolation patch with custom curves. And it took a lot of time. Timelines makes it a truckload easier to choreograph your objects for this kind of motion.
  • Timelines in Quartz Composer

    The new Timelines patch allows for arbitrarily complex curves. And Stuff.

    Working with Images

    • Sprite, resized: It’s now a breeze to display sprites at the original size of the graphic – and with pixel alignment, if required. In Tiger, it took quite a number of patches to make sure that the image was resized in proportion to compensate for the window, proportional resizing to result in a pixel-for-pixel output. Did you get all that? Now it’s a pop-up menu option. Ed.: Translation: you don’t have to worry about sprite resizing any more!
    • Anti-aliased edges on sprites! Yay, rotating rectangles no longer need to look like they are from the 80s — although spheres and cylinders still sometimes do. Which reminds me, although we now have cylinders at out disposal, a few more generic shapes would be handy. Ed.: There was briefly an OBJ import for bringing in more complex objects, but unfortunately, if you really what this kind of 3D you’re going to need to go to another tool. I like free utility Processing for those jobs; see processing.org.

    Anti-aliasing in Quartz Composer

    Switch on the ‘anti-alias’ checkbox to be transported to this decade
    • Anchor Position: makes it dead simple to calculate where images need to go to stay aligned with the center or edge of a window.

    • All-Pixel Access: You now have access to pixel values within your images – you can find out what is the color value of point XY, or the value of an average area. This opens up lots of opportunities for changing data based on an image, basic object tracking, and the like. Ed.: this will at least allow some of the simple motion tricks previously performed in tools like Flash 8 / Flash 9. Very cool.
    • iSight Control: There are new controls over the iSight input, letting you switch off auto-exposure, which is important for tasks like green-screening.

    Input

    • Joysticks, gamepads, and more: A new HID input patch means you can now use any HID-compliant USB device to control your Quartz Compositions — great for connecting unusual USB hardware to control your video creation.
    • Apple Remote: The new Apple Remote patch lets you listen for button clicks coming from an Apple Remote.
    • Data crunching: Quartz Composer can now load, and download XML files, which makes it much easier to move large chunks of data in and out of your composition.

    Output

    No more silent movies: The Movie loader in Quartz Composer now supports audio out – flick the switch for Asynchronous Mode. This is a handy to have, but it does mean the Quartz composer timebase is ignored, so some more testing will be required.

    Communications

    OSC and MIDI: Leopard brings a new OSC (OpenSoundControl) client and server patch, as well as improved MIDI support which will excite many CDMers. Ed.: MIDI support means you can plug in keyboards and control surfaces and support that more easily. And OSC opens up other options, like use with the Monome open-source controller hardware. See the OpenSoundControl site … and, really, nothing about it really is specific to “sound.” Open Control would be better. -PK

    Multiple screens — or multiple projectors: There is now support for running Quartz compositions across multiple screens, and also a cluster. From the Release notes:

    There’s a new Quartz Composer developer tool located in /Developer/Applications/Graphic Tools: Quartz Composer Visualizer. This application can render a Quartz Composition spanned over multiple screens on a single computer or even on a cluster of computers. The source code of Quartz Composer Visualizer is available as well in /Developer/Examples/Quartz Composer.

    Updated: Here’s a look at some of those hidden patchers in QC.

    Quartz Composer hidden patchers

    The Future

    Overall a great update. Sources tell CDM that there is a stack of ‘Private’ patches left in there, which didn’t make the cut for Leopard. So we may still see even more interesting patches from Apple in a dot release. Third-party patch developers, such as the well-respected QC guru Sam Kass, are excited by the possibilities that this new version of Quartz Composer offers. In addition, Apple is now encouraging the sharing of Compositions between applications, so this may encourage more developers to get on board the Quartz Composer locomotive. For the non-developer, there’s a bright white light shining at the end of the tunnel, the possibility that one day, tools like Quartz Composer, Automator, and Core Animation may be combined to let the average mom and pop knit together simple, yet stunning, software applications for their own personal needs. There’s no question that Quartz Composer, like many patching environments, is difficult to manage when used to build bigger projects. Making structural changes is still a chore, requiring much manual rewiring, and QC still lacks any simple graphical way to build logic systems, or complex or dynamic processing structures. Other Cons include a steep learning curve, which has only been slightly bettered with the new templates, and a clumsy interface. Therefore, with Leopard, Quartz Composer will remain a niche, if not more powerful, eye-candy tool.. but overall, for types like us, Quartz Composer is a really cool tool that is well worth the time getting to know.

    Software knitting!

    This is not your Father’s computer. It’s your Mom’s.
  • CongoZombie

    The new Quartz Composer is pretty good- I've VJ'd with it already, and I think that it will be great when people start releasing new patches/plugins (now it's easier to produce them).

    I'd like to see a good tutorial on creating cocoa interfaces for compositions (because I don't know about anyone else, but I was somewhat confused by apple's documentation on the issue…)

    I'd like to see support for properly deforming the 3D shapes, because I'd like to do some glitchy stuff.

    I'd like to see quartz composer develop to maybe compete with something like VVVV. (I'd be using it now if Leopard's bootcamp actually worked like the Tiger beta did… is it too much to ask to actually be able to install windows?)

  • CongoZombie

    Wow, I sure said 'I'd like' a lot there

  • http://danwinckler.com Dan Winckler

    CongoZombie,

    Check out Performer, an open source application that is a simple Cocoa wrapper for QC comps (with some other handy features): http://10base-t.com/premier.html#performer.

  • http://axiom-crux.net nico

    how do I get this?

  • http://songcarver.com Keith

    Nico,

    You need to install the Developer Tools which comes with Leopard. It's a separate install which you need to do. After that, have a look under Developer>Applications for Quartz Composer.

    Compositions that you make can be exported to Quicktime which actually render the composition in realtime (on any mac with 10.4 or later.) This means you can share your creations with other mac users without them needing to have Quartz Composer installed.. and it's a much smaller file. however, if you want to render it out, frame by frame, then you can drag it into iMovie and export as a normal Quicktime file. This has the advantage of working for any windows/ mac user with QT installed.. but the disanvantage that the file will now be much, much larger.

  • CongoZombie

    Thanks Dan, I'll give it a look. I was just VJing with some quartz generative stuff, and the club owner told me it was "boring" and it looked like "random stuff that a computer would make"

    I think that's it guys! Clearly us VJs are out of a job, because computers can just 'make' visuals.

    Guess it's back to isadora for some clip based stuff (yawn)

  • http://createdigitalmusic.com Peter Kirn

    For performance, I don't think you'll beat the combination of VDMX + Quartz Composer … because VDMX does all the handling of video files, mixing, etc., and works with QC and live video input, as well. We're just waiting on a more finished build at this point. It'll be worth it.

    I love club owners. They're always such a useful font of aesthetic wisdom. ;)

  • http://abstrakt.vade.info/ vade

    GLSL !
    GLSL!
    GLSL!

    ZOMG.

    How was that overlooked. GLSL! ZOMG! Say it with me. OpenGL Shader Support within Quartz Composer = XBOX HUEG

  • CongoZombie

    VDMX- have you found that it really matters which layer you put the quartz composition in? Because I've found that if you put a video underneath a quartz composition (or the other way around, I forget) it causes the video to stutter- I can't see that it's my computer (brand new macbook pro)…

    Other than that though, quartz and it are made to go together.

    Oh yeah, I almost forgot- some of my leopard quartz patches just show black in VDMX :/

  • http://createdigitalmusic.com Peter Kirn

    I know the VDMX support is very much in-progress — there are some really, uh, interesting coding issues that happen working with QC. I have gotten the QC stuff to work in different layers. I think we'll just see this stuff ironed out in future builds. But to me, VDMX is the one app in which I could see really getting serious about QC in performance. (for QC, anyway — there are other, non-QC or non-Mac options, too!)

  • http://www.resolume.com bart

    Quartz Composer 10.5 does FreeFrame too!

    The developer examples in 10.5 include a FreeFrame host plugin for Quartz Composer.

    This means you can use FreeFrame plugins in Quartz Compositions!

    Instruction on how to compile and use it on vjforums:
    http://www.vjforums.com/showthread.php?t=22996

  • Pingback: Quartz Composer in Leopard - The Apple Blog

  • http://allanwhite.net Allan W.

    Very cool stuff. I'll be watching.

    I do a lot of worship gigs – the ProPresenter people (a great video playback tool, BTW) hinted at more QC integration to come – enabling (theoretically) and all-in-one worship media and VJ tool (potentially driven by audio or MIDI devices). Throw a touchscreen controller in there and I'm in, well, Heaven. =)

  • Marcus B

    Are patches created in Leopard backwards compatible with Tiger?

  • http://blogs.walkerart.org/newmedia/ justin

    I am very excited by the new QC. Quartz Composer Visualizer is a godsend if you're looking into rolling your own digital / dynamic signage. I did a little playing with it for that and wrote up a little post about it.

  • http://www.songcarver.com SongCarver

    Marcus B

    Only patches which use the Tiger compatible objects are backwards compatible. There is a function included in Leopard's Quartz Composer to test patches for backwards compatibility.

  • http://wileywiggins.com Wiley Wiggins

    Looking forward to getting into the new QC… I wish a leopard-specific book or tutorial were available.

  • Pingback: Create Digital Motion » Quartz Composer Tutorial: Lighting 3D Cubes and Moving them with Audio Input

  • http://www.jonbro.tk JonBro

    Yeah, I gotta second vade there, the support for GLSL shaders in QC is totally awesome. With glsl support and a bunch of time you could theoretically build any shape you would like, as well as doing all kinds of convolution kernel stuff really, really, really fast.

    That said, so far all I have built is a kinda crappy blur fragment shader, but I am pretty excited about it.

  • Daniel

    So how do I get the color of a given pixel as described in this article, and how can I do the same thing in the Tiger version of Quartz Composer?

  • http://www.songcarver.com Keith

    Daniel,

    There is no low-level access like this in Tiger. In leopard the patch is called "Image Pixel", and there are other patches which lets you get the average color over a certain area etc.

    Hope that helps!
    /Keith

  • Pingback: links for 2007-12-31 at dpwolf/blog

  • Pingback: Muzyka » Blog Archive » New on CDMotion: Scratching Vinyl, Coveting Toys, Plugging and Playing a Visual Jams

  • http://airvent-media.co.uk arctic-sunrise

    sorry t sound dumb but whats the windows xp equivalent of this

  • Pingback: KINECT #1 « Digital Dump

  • nikki

    this was kinda very boring! sry

  • http://google.com Julio Sawin

    Awesome job!