Gesture to Gesture: cross-modal gesture recognition

Gesture 2 Gesture is a visual instrument that maps the sound gestures to freehand drawings. Made for Claudia Schmitz.

Before the performance, artist trains the system by recording sound and hand drawn gestures at the same time. One ‘gesture’ can consist of many recordings to catch all the variations.

During the performance, artist produces the sounds and the system tries to mach it to the hand drawn gestures, producing a live painiting.

To make this system work I used Ircam’s excellent Mubu library, and the painting code is all done in Jitter. Every paintstroke is recorded, and the gaps between the mouse/pencil driver reports were filled automatically to produce continous smooth lines. The painting system also randomized brushtip size and rotation to produce more expressive gestures. The resulting image is sent to Resolume where it was further processed and projected.