Applications of gestural sensors & interfaces to musical controllers
Introduction
This page contains applicative examples of the gestural interfaces and sensors digitizers built by Emmanuel Fléty / PIP between 1997 and present.
Foreword & Background
Timeline of work
Various gestural controllers and interfaces (move later to dedicated page)
Bibilolo - Marc Monnet & Percussion de Strasbourg (1997)
Custom analog to MIDI interface for 32 FSR sensors with velocity extraction (E. Fléty).
| Keyboards | MIDI Interfaces |
|---|---|
 |  |
Machinations - Georges Aperghis (2000)
MIDI interface for mechanized light diaphragm (Iris) to control video projection outputs
Eluvion-Etude - Lucia Ronchetti (1997)
MIDI ultrasonic sonars used to detect the attitude of a camera-woman filming the instrumentalist (far-near / high-low PoV) ** add link to report **
S0undB1its - Robin Minard (2005)
S0undB1its is an installation for 576 piezo speakers (paper cone) installed ivy-styled in large rooms or an empty swimming pool. Each speaker uses 1-bit audio consisting of PWM signals generated at audio range sampling frequency. Individual speaker signals are extracted from 24 bit ADAT formatted digital audio sourced by an RME sound card with 3 ports resulting in 3 x 8 x 24 = 576 speakers. PWM signals produce clicks, buzzes, squeeks and sweeps spatially distributed. Custom ADAT decoder & splitter + amplifier : Emmanuel Fléty - Max/FTM audio matrix & 1 bit source combiner : Rémy Muller 
Emilie Simon's voice controler "B.R.A.A.H.S" (2003 version)
Emilie's first arm controller used a custom wireless analog to MIDI unit developped by Emmanuel Fléty in 2001 
captation vidéo Eyesweb Hervé Robbe + Andrea Cera
Cirque du Tambour - 3D Glove
Anthem 2 (travelling / Camescope) Cadre interactif / Raffinot (interface dédié IR cf article) - Al Segno
Ofelia -Lisa Streich (2022)
OSC based (wired Ethernet) motor controller for prepared piano. Real-time control of motor speed and direction driving leg-shaped paper legs over the piano strings.
AtoMIC Pro (1998-2002)
Background
AtoMIC Pro (Analog to MIDI Converter) is a 32 input analog digitizer able to convert sensors signals to configurable MIDI messages. It embeds a LCD and a small keypad for onboard configuration and can retain 32 diffrent configuration presets. It's also equipped with 8 digital inputs and 8 digital outputs (those can be controled via sysEx messages)
T-Stick - Andrew Stewart
FSR Augmented T-stick + Paper FSR's on the floor
OMNI - Patrice Moullet - 4 AtoMIC Pro
The OMNI (Objet Musical Non Identifié - Un-identified Musical Object) is a digital percusive instrument composed of a dome equiped with 108 piezo sensors. After a being rectified and amplified by a custom analog front-end by Jean-Loup Dirstein, velocity sensitive MIDI notes are generated using 4 AtoMIC Pro units.
 |  |
Stretch Machine - Patrice Moullet - 1 AtoMIC Pro
The stretch machine uses Hall effect analog sensors installed on a lever with a magnet and amplified by Dirstein's electronics. Paracord strings operate 8 levers when pulled, their signals are further digitized by an AtoMIC Pro unit. MIDI messages are split between 4 voices note-on generation, and 4 continuous controllers to alter the sound (pitch, timbre).
 |  |
Eobody (2003)
Background
Eobody is the little brother of AtoMIC Pro with 16 analog inputs and some added controls on the front panel (3 rotary potentiometers & 4 switches) Designed along with french company EOWAVE and distributed by them.
Passage - Pierre Jodlowski & Christophe Bergon (2009)
Passage is an interactive installation using IR proximity sensors digitized to scan the visitor while walking thru a corridor where sound memories are triggered
Ethersense (2004)
Background
Ethersense is a modular sensors to OpenSound Control interface. The standard version is composed of one OSC converter mother board and 2 x 16 bit analog-to-digital daugther boards (16 inputs each).
Jodlow-Pads - Pierre Jodlowski
Jodlow-Pads is a high resolution / low latency percussion interface doubled with pressure control often used in combo with a Wacom tablet and stylus (2D orientation).
 |  |
- Music Violence (2008)
- Cinématèque de Toulouse (2009)
- Images d'une Oeuvre (2011)
- Trio PAJ (2011)
- Avant Art Festival (2020)
- Trio ACJ (2024)
Mendel - Pierre Jodlowski (2005)
Interactive installation "fed" by the audience interaction and gestures. 
Bogen Lied - Florence Baschet (2009)
Augmented Violin prototypes 1 & 2 using the Ethersense and custom daugther boards (Electric Field Sensing and wireless IMU).
Thinking Things - Georges Aperghis (2017)
FSR based helmet to trigger sound grains 
WiSe Box - WiFi Digitizer (2005)
Background
The WiSe Box is a WiFi OpenSound Control sensor digitizer. It converts 16 analog signals to 16 bit OSC messages wit a sample period as low as 3ms. Wireless combined with low latency and portability allowed for creating multi-peformer pieces especially choreographic works.
This is my house - Myriam Gourfink (2005)
Choreographic piece for 5 dancers equipped with custom optic fiber bend sensors placed on shoulders, IMUs and a respiration belt sensor, all digitized and transmitted wireless in OSC to the computer. Cyclicity and synchronicity of move / breathing detected by autocorelation (Rémy Muller) and generating real-time constrained Laban score for the dancers. Choreographic phrase detection based on Hidden Markov Models (HMM), a foundation study work for the forthcoming Gesture Follower. 
Double Points - Hans Peter Kyburz - Emio Greco (2005)
Flexion sensors integrated in the dancer's stage costume, digitized by the WiSe Box and driving electronic music in real-time.
I will not kiss your F.ing Flag - Marco Stroppa (2005)
Augmented Trombone with a wireless IR ranging sensor measuring the continuous change of the slide.
Le grand Dehors - Pierre Jodlowski / Emmanuelle Huynh (2006)
The WiSe Box is digitizing sensors from a box appearing on stage as a transitional object. When held, metal electrodes on its faces measure galvanic skin response of the body. An accelerometer is also digitized to provide dynamic interaction with the box. 
Maya the Bee (2007)
INFO
Maya the Bee or "Bees" modules are XBee (Zigbee) based wireless digitizer considered as our first hybridation between sensor interfaces and modules. The Zigbee protocol was used in placeholder until Wi-Fi based MCU with proper & efficient 802.11 modem were available. XBee RF modules are paired from the emitter to the receiver on a fixed ISM band channel (2.4 GHz) and further converted in OSC (wired Ethernet) by the base receiver station
| Bee Transmitter | Bee OSC Receiver |
|---|---|
 |  |
StreicherKries - Florence Baschet (2007)
Augmented Violin 3rd prototype for the piece StreicherKries from Florence Baschet and Quatuor Danel. Bees (Zigbee) modules with analog 5 axis 10 bit IMUs (3D Accelerometer & 2D Gyroscope).
MO - Modular Objects (2008-2011)
Background
MO (Modular Objets) are gestural controllers elaborated with designers during the national research project (ANR) Interlude and explore alternative paradigms for music pedagogy. They features a low latency wiress MCU bound to a Zigbee modem and a 9DoF digital IMU. Various accessories can be plugged to the main "remote" such as piezo pickups or a conducting wand.
- Project
- Project
- Demo
- MO-Kitchen - Guthman instrument competition (2010)
- Exhibit St Etienne (2010)
- Exhibit MoMA "Talk to me" (2011)
- Exhibit MoMA (2011)
- Prized at Digital Meetup (2013)
mini-MO & MO-Nano (2011-2015)
Eigenspace - Mari Kimura (2011)
MO-Nano on Glove - Augmented violin & interactive graphics
 |  |
LunaPark - Georges Aperghis (2011)
Gloves + IMUs triggering sounds along with gesture recognition
Your Fox's A Dirty Gold - Alexander Schubert (2011)
Urban Musical Game (2013)
Urban Musical Game is a sportive interactive ball game using street-soccer foam ball augmented with the mini-MO (Zigbee) wireless IMU. Gestural recognition such as spinning, freefall, energy and impact as well as sound mapping is achieved on the computer. Goals are detected by other mini-MO units installs in nets hung in a barrel where the ball has to be thrown in to mark. Sound design & Composition by Andrea Cera. 
Sinekine - Greg Beller - Richard Dubelski (2014)
Arboretum : of myths and trees - Diana Soh (2013)
Serious Smile - Alexander Shubert (2013)
Gesture triggered electronic percussion using IMUs
Stonic (2015)
Stonic is an object equipped with a MO-Nano module doubled with a piezo analyzer extracting frequency ranges, in the same spirit and architecture of the Modular Objects MO.
Dels Dos Principis - Henri Foures & Jérome Thomas (2015)
Interactive juggling using wireless IMU in augmented balls. 
Musical Ping-Pong (2015)
Mo-Nano modules installed in custom Ping-Pong racket handles and augmented table. Also used for research to study the expert gesture of Ping-Pong practitionners in the context of the Legos research project hosted by the IRCAM ISMM Team. 
R-IoT - 9 DoF WiFi IMU (2015 - Present)
Background
The R-IoT (Responsive Internet of Things, but could also stand for Rachmaninov) is a miniature WiFi OSC IMU. It features an Arduino compatible MCU and a low latency WiFi modem along with BLE in its latest revision (v3 - 2025). placeholder for a dedicated R-IoT page link. Powered by a small Li-Po battery, it has a runtime of > 3hours. It achieves IMU / MARG sensor fusion to provice raw motion data plus 3D orientation under the format of Euler angles and Quaternions
Music Bricks & RapidMix Projects (2015-2017)
The first generation of the Wi-Fi IMU wireless R-IoT was field tested by hackers during Music Festivals and Hackathons, a 24hr marathon of hacking. The module was presented to the participants and some applications built during the hackathons are presented below
- Music Tech Fest (Umea) Hackathon
- Music Hack Days (Barcelona) Hackathon
- Interactive video cube
- Hi Note - Paraorchestra
- GIRD - Interactive Dance Floor - Tracy RedHead)
- Interaction Tools for hackers and artists - Conference (2017)
Introduction to the R-IoT - RapidMix Project(2018)
FreeMIX - Joseph Larralde (2015)
Triggering and controlling Freesound samples using gestures (Rapid-Mix project)
GeKipe - José Miguel Fernandes (2016)
Space distributed percussion and generative video using 2 R-IoT modules (wristlets) and a Kinect
Homotopy - José Miguel Fernandez (2016)
2 x R-IoT gloves for live electronics
Mad Max - Pierre Jodlowski & Philippe Spiesser (2017)
R-IoT gloves - in the legacy of GeKiPe
La princesse Légère - Violetta Cruz (2018)
Interactive stage element : swinging board augmented with IMU + Interactive Props 
IDEA - Sampo Haapamäki (2018)
R-IoT wristlets to extract, analyze and archive the conducting gestures dataset. Real-time tracking + sync during the performance
HeartFM - Elaine Chew (2020)
R-IoT used to sense pianist attitude and respiration
The perfect Anti Object - Damian Gorandi (2021)
R-IoT Gloves for live electronics and percussions
Frammenti di memoria abolita - Rachel Beja (2021)
R-IoT module installed on a Olive tree wooden board + FSR sensor, used a musical controller
Rosebud - Matthias Krugger (2022)
2 x R-IoT Gloves + 1 x Headband for dance and live electronics
H2O / in memoriam A-68A - Basile Chassaing (2022-2023)
Wrist-worn R-IoT controlling live electronics and stage lighting
Day 0 - Trans-Instrumentalism - Satchie Kobaiashi (2022)
2 x R-IoT gloves controlling live electronics + OSC controlled light bow
Como-Rehabilitation - Iseline Peyre (2022)
Two R-IoT modules used as limb motor control rehabilitation (post stroke) thru sound feedback
Entropie - Compagnie Zimarel & Léo Lérus (2023)
Source Rayonnante - José Miguel Fernandez (2023)
Gesture Following using ISMM GF objects and recorded training to feed Antescofo score following
Tsukumo-café - Juliette Séjourné - Frédéric Bevilacqua - Laval Virtual (2023)
Interactive coffee mug
- Piece
- Publication
- [Excerpt] (https://vimeo.com/939049615)
SPA[S]M - Basile Chassaing [Work in progress] (2024)
4 x R-IoT installed on dancers wristlets to control real-time electronics