Human Visual Instrument: Interactive Immersive Visuals

Last week a friend of TripleWide Media (Nicholas Rivero) shared with me a link to one of the coolest video projects I’ve seen yet. I knew I had to share it with our community. We’ve talked about media being used as a vehicle to communicate in the context of an environment… much like an instrument would in a band. Art adds to the collective of anthems to create energy, reveal truth and add a soundtrack to the moment.

However these guys have taken it to a whole new level… they’ve figured out how to make video into a virtual musical instrument. Here’s what I’m talking about:

The project has been a long time in the making, as it apparently took months to put the whole thing together. Using the power of Ben Kuper’s BiKinect project, they morphed a Microsoft Kinect box into a musical instrument with the addition of Abelton Live.

The Music and video however were indeed two separate systems to prevent lag wherever possible. Blogger Jeff Nusz at posted this diagram to show how the two Kinects were being used through a Mac and PC to drive Audio and Video respectively.

System Design for Video Instrument System


With the use of the Kinect, the person controlling the musical instrument is able to use every day hand gestures to control sounds like a keyboard. In this video you can see how they simply move their hands in space to change the key being played…

Related Article:  The Creative Conference You Need to be At!

IN the process of creating this instrument, they used a plethora of various apps to develop and debug their system. Here is the list:

1. Keyboard mapper – AIR app that is used to modify and edit keyboards on the fly.
2. Mapping Tool – Custom Processing app to speed up the process of mapping movements to midi and Ableton.
3. Midi Testers – Sender and receiver to help test system connection.
4. Midi Interceptor – Testing app that intercepts midi from the system helps in trying to figure out where issues lay.
5. Connection Testers – Data send emulator to test data receivers.
6. Motion Management Tool – custom AIR app that provided an interface to help create and manage positions and movements

The Visuals

Of course as we know here on TripleWide Media, the visuals are just as important to the overall experience. Don’t get me wrong, they were a critical piece of the puzzle for this project as well. As you can tell in the lower concept render of what they wanted this to look like, you can see that the “green polygon man” was a theme from the beginning. I think this helped make you realize that he was controlling the music live instead of the “disney-trick” of having the actor/dancer/musician being well choreographed with the moves that needed to be pulled off.

Early Concept Render of Project

Layers of Musical Instrument

This above sketch is a drawing from Jeff ( showing how they layered all the various visuals in place. Everything was rendered out (for the landscape layer) at 1920×768 64-bit Motion JPEG Quicktime videos at 60fps. That’s a heavy codec to have to deal with.

Related Article:  Live Event Tools: Datapath X4

Other visuals included the polygon man:
Polygon Man - Human Musical Instrument

User Interface for the Man to interact with:
User Interface & Design

How did they do the projection you ask? With 2x HD-20k’s from Christie, edge-blended using a Vista Spyder system. The overall projection surface for the final project was almost 100 feet wide.

For more details on this entire project, check out Jeff’s posts at Custom Logic or Paul’s post on The Fugitive. Links here:

Custom Logic: V Motion Project: The instrument

The Fugitive: How We built it.

Custom Logic: The Visuals

Luke McElroy

Author Luke McElroy

Luke McElroy is the founder of Orange Thread Media, the parent company to TripleWide Media, SALT Conferences and Orange Thread LIVE. He is the author of The Wide Guide: Blueprint for the Multiscreen Movement. Hailed as one of the “top innovators for worship” by Worship Leader Magazine in 2013, Luke’s leadership has helped create powerful worship environments for thousands of Church communities throughout the entire world. He currently lives in Nashville, TN and regularly writes about creativity, leadership and faith at 

More posts by Luke McElroy

Join the discussion 3 Comments

Leave a Reply