Imagine what’s now possible with a mobile phone: anyone with a supported device can jam with other artists, walk up to an installation, connect to other creators and other software, all using supported protocols. Leaving behind the days of painstaking manual adjustment of MIDI commands and obscure drivers, and even the act of having to physically connect gear, software - and with it, digital art - can simply talk to each other in standard ways.
That’s why we’re excited about software like Memo Atken’s MSA Remote. It uses the standardization provided by the network-savvy, open protocol OSC, with additional plug-and-play (or, erm, don’t plug, do play) functionality from the TUIO protocol. OSC provides the communication; TUIO makes the messages standardized.
To avoid confusion: You do NOT need a Mac to use OSC. OSCulator is a cool app - and makes bridging to MIDI easier - but it’s just one tool among many. You can use this app with Windows and Linux, too, and visual apps like VDMX, Resolume Avenue, Pd/GEM, Processing… the list goes on. In fact, almost every visual app today worth using uses OSC, even as the music world is painfully slow to catch on.
When it comes to controlling software, let’s put it bluntly: OSC good, MIDI bad. With OSC, it’s possible to control the array of things software might do, with easy use of high-resolution data, descriptive names in plain English (or your language of choice), a path hierarchy that makes it easier to structure messages in modular software, and smart networking features that makes assignment and communication a breeze. With MIDI, um… well, prepare for lots of mucking around.
Happily, visual software developers proprietary and open source alike have done what music developers generally haven’t - embrace OSC. Thanks to the fact that this community is unburdened by tradition and commercial development tends to involve small, responsive teams, change hasn’t been so tough.
So, visualists, it’s time to reap the fruit of that development work, and make the live performance rig work the way you’ve always dreamt it should work. Our friend Gian Pablo (check out his fantastic blog) clues us in to some recent developments with Resolume 3 “Avenue.”
We have the tools. We have the techniques. Now, what happens when you put technology for tracking physical objects into the hands of artists around the world?
On June 6, members of the CDM community joined in our first “global hackday,” assembling tangible interfaces on tracking tables. Martin Kaltenbrunner of reacTIVision and the reacTable joined us from Vienna, Austria, while Adam Kumpf of the OpenFrameworks-powered Trackmate and MIT Tangible Media Group chatted from Cambridge, Massachusetts.
Much of the day was about saying “hello, world,” and helping each other through getting cameras working, troubleshooting, and the like. But there was an extraordinary variety of ideas even in one day. I can only imagine where things might go from here. I can also see the tools people were developing as being expressive for live visuals and visual performance - and via a really cheap interface, too.
Some of the accomplishments of our first day:
Endless ideas: Drawing interfaces using objects, a floor tom as a housing, a musical instrument with soda bottles as the interface, a game with blocks featuring the Tokyo skyline, and others.
Troubleshooting data: Both the Trackmate and reacTIVision projects got lots of feedback about how people were using the tools, where tracking was and wasn’t working, and where people got stuck up. We also compiled lots of information on cameras, drivers, builds, and operating systems. I’m working with Adam and Martin to dig through a lot of this information so we can compile a really practical guide to make it easier for people to create their own projects.
Special guests: Marco had his augmented magic show and we had the beginnings of an interactive glove. Check out the video highlights to see what the NYC event was like.
noisepages for networking, and other tools: Livestream video was a bit of a mixed bag; I’m still looking for easier ways of doing that (both on the video shooting side and the streaming side.) Text chat was an easy win, though IRC can still be cumbersome; I’m looking into integrating standardized XMPP chat instead, and providing access via any client or a webpage. But the other big success story was that noisepages worked nicely for documentation; see the fritzcrate and i3games sites for great examples!
Building Communities Around the World
A real highlight to me was getting to hear from Valeria (jalea.tv) and Jose (Estado Lateral Media Lab), visiting New York from Argentina. They talked about what the scene is like in Buenos Aires, and touched on issues of community, learning, open source, and the multilingual world of coding. And they do some really beautiful and hip visual work, both commercial and experimental. We also wound up with a significant amount of the online chat being in Spanish. That to me is a healthy sign - “global” really doesn’t necessarily translate to “English.”
Hackday Results, and the Future
For more documentation, head to our noisepages site:
And, yes, this is only the beginning. My suspicion is that a single weekend would be enough to get workable tracking projects going - especially as we iron out some technical wrinkles. But we’d certainly love to do more of this, whether it’s another “official” hackday or simple an open lab with chat and sharing.
noisepages is still in alpha state, but we’re working aggressively to move forward to beta, and content placed there is safe and future-proof. (Most importantly, I’ve fixed the jpeg library on the server so that avatars work!)
It’s been a long time since I touched Livid’s VJ software. But when it comes to hardware, they’ve been doing unbelievable work that could have Resolume and VDMX and Processing and vvvv and Max users turning their heads.
The Ohm64 looks simply fantastic. It’s got an ideal configuration for a lot of live visual applications, with 8×8 triggers and plenty of faders and knobs to go along with them. It’s also finally a controller that’s well-made but doesn’t cost a fortune – you get a well-crafted device made by the people who designed it in Texas, but at $599, it’s still affordable.
Visualists are doing all kinds of new things to expand their performance, so I believe having a truly open controller is essential. The Ohm64 delivers, with a chip and editor software that have extensive open support. That means that, as with the brilliant monome controller, you should see a community that experiments with creative ideas for how to use it. (Nor do I think this is necessarily monome competition – the monome is still beautiful for its minimalism, whereas this should appeal to people who ignored the monome because they needed knobs and faders for additional parameter control.)
For visual software increasingly using OSC, a future firmware update should provide native OSC support (and possibly even DMX in the near future). For everything else, there’s MIDI support now. And unlike the Akai APC40, that means real MIDI support, with actual MIDI in and out ports and endlessly customizable controller assignments and LED feedback, instead of the Akai’s single USB port and permanently-fixed layout. And this is fully bus-powered, so you’re not screwed if you forget your power brick.
I did a full preview for CDMusic. But next week I should get to try one in person, which is the real test.
reacTIVision is able to work thanks to these alien-looking markers called “fluidicials.” Karsten Schmidt aka toxi has developed software for creating more of these markers, and it generates characters like this “teddy bear” seen here.
Computer vision for tracking movement is cool. But add the ability to track actual objects, and you can extend the possibilities for interfaces. We’ll be playing around with this on our upcoming tangible hackday. A lot of the reason these experiments can proliferate is the availability of free frameworks that make the technology accessible to artists and designers. The tricky tracking work is done, leaving you to focus on where this tracking might actually be useful.
The other good news: while doing projected visual feedback or fancier tracking can get more complex and costly, if you just want to track some objects, all you need is a USB or FireWire camera and some printed stickers. Cost: $40 with a webcam, about $5 without.
Recently, one of the most popular of these libraries got a big update: reacTIVision 1.4. It’s the open-source, multi-platform framework that powers the reacTable, and was developed by Martin Kaltenbrunner and Ross Bencina at the Music Technology Group at the Universitat Pompeu Fabra in Barcelona, Spain.
Multi-platform really means multi-platform. It works with FireWire and USB cameras, Mac, Windows, and Linux, and has clients for C++, Java, C#, Processing, Pure Data, Max/MSP/Jitter, Quartz Composer, and Flash, plus a wide range of applications that support the OpenSoundControl-based TUIO protocol.
I’ll actually be testing both the tracker in reacTIVision and the Trackmate tracker from the LusidOSC project. The Trackmate software is built in OpenFrameworks. It does use a different protocol (LusidOSC), but that’s also based on OSC, and there’s even a tool that translates to TUIO.
For an example of what this all looks like when assembled - and some of the power of having a framework on which to build - here’s a tangible interface for a multiplayer game. It’s Pong with objects.
This games uses reactivision software, along with Flash, to detect symbol fiducial block movements. The game is played by moving these symbols on a table. Players can enter and exit the playing field at any time. The game adapts to the number of players. The lower the score the better; the first player with a score of 12 ends the game.
Music: Waterdrops by Yohan Shin http://www.geocities.com/cerup2
There’s more progress coming in reacTIVision world, too. First up: reacTIVision 1.5. Martin tells us:
After this release I am now implementing reacTIVision 1.5, which will improve the multi-touch tracking performance, and already implement the upcoming TUIO 1.1 blob tracking extensions, for the transmission of basic untagged object descriptors.
Following that, the next plan is for TUIO2, an expanded protocol that will address some of the shortcomings of the first version, to be released with a future update to reacTIVision. You can read the full specification for the new protocol, but Martin has kindly given us a Cliff Notes version:
To summarize, TUIO2 has a flat profile, which now includes symbol, cursor and blob descriptors in much more detail. Symbols now also can carry content info (e.g. datamatrix), cursors have additional properties such as type, pressure and region of influence, and blobs can be described in various incremental messages that describe the bounding, contour and skeleton for example.
The other important thing that Martin is doing - and the reason for the wait - is to synchronize implementation of TUIO in other key libraries and clients. That is helping keep TUIO a standard for this kind of work. It’s not even really a full protocol - part of the beauty of it is that it builds on OSC.
If this isn’t quite making sense yet, stay tuned and we’ll show some of the specific applications and get you started with your own projects.
15
