Embedded Files
Week05
Gesture, Sound, and Time-Based Interaction in XR
Gesture, Sound, and Time-Based Interaction in XR
Designing media that unfolds through the body over time: hand tracking, temporal behavior, procedural media, and spatial acoustics.
Opening Discussion (6 min)
You have explored N-body systems through computation, flocking through simulation, and the Solar System through data.
What could you create by adding a layer of interaction?
Similar code, but all of us choose different moment.
01 · Core idea
Interaction is a temporal relationship.
Interaction is a temporal relationship.
A gesture is not a button press.
A button triggers an event. A gesture shapes a process.
In XR, the hand gives us much more than a binary “click.”
Position, velocity, acceleration, orientation, openness, pinch strength, bimanual distance, repetition, stillness, and trajectory all evolve over time.
The artistic opportunity is to interpret those changes as behavior.
These changes happen over time. That temporal structure is where gesture becomes useful as a medium for interactive art.
HAND → MOTION → TIME → BEHAVIOR → MEDIA → SPACE
Do not ask only:
“What parameter should this hand control?”
Ask:
“What kind of behavior should this movement create?”
02 · Time as a medium
Design what happens before, during, and after the gesture.
Design what happens before, during, and after the gesture.
A single frame gives us a pose.
A sequence of frames gives us movement.
A history of movement gives us rhythm, momentum, repetition, anticipation, memory, and decay.
Position can become a trajectory.
Position can become a trajectory.
Instead of using hand position only as an XYZ control value, preserve the path.
A gesture can draw, carve, conduct, push, orbit, or leave a trace in space.
Velocity can become energy.
Velocity can become energy.
A fast movement can inject energy into a virtual material.
That energy may drive sound density, particle activity, brightness, instability, or resonance.
Acceleration can become impact.
Acceleration can become impact.
A sudden change in movement can trigger accents, bursts, collisions, or sharp changes in sound and visual intensity.
Stillness can also be meaningful.
Stillness can also be meaningful.
When the hand stops, the media does not have to stop. It may ring, decay, reorganize, drift, or remember what happened.
Repetition can become rhythm.
Repetition can become rhythm.
Repeated actions can create pulses, patterns, synchronization, accumulation, or gradual transformation.
The most interesting variable is often not the hand itself,
but a state that the hand changes over time:
energy, tension, density, charge, instability, memory, or flow.
03 · Hands are not controllers
Use the body as a continuous, expressive input.
Hands should not simply imitate controllers.
Use the body as a continuous, expressive input.
Hands should not simply imitate controllers.
A pinch can function like a click, and sometimes that is exactly what we need. But if the central interaction is only “pinch to activate,” much of the expressive potential of hand tracking is lost.
Example of Weak Translation
Example of Weak Translation
Pinch → Play Sound
This is useful UI, but it is still essentially a button.
More expressive translation
More expressive translation
Approach → sound emerges.
Acceleration → energy is injected.
A sweeping trajectory → a field is shaped.
Release → the result continues to decay.
Returning later → the previous action changes the new response (modify memory)
+ Tracking is not neutral.
Design for jitter, occlusion, imperfect finger recognition, variable speed, and loss of tracking. Favor tolerant continuous behavior over interactions that require a single exact pose at a single exact frame.
04 · From mapping to behavior
Think beyond “control.”
Think beyond “control.”
Design an interaction model, not a parameter spreadsheet.
Design an interaction model, not a parameter spreadsheet.
“Hand X → pitch, Hand Y → volume” is useful for learning, but sophisticated interactive work usually inserts an intermediate behavioral model between sensing and presentation.
Direct mapping
Input → Media Parameter
Input → Media Parameter
hand Y → pitch
pinch → play
speed → volume
Immediate and readable. Good for prototypes.
Often becomes literal or “instrument-panel-like.”
Behavior mapping
Input → State → Media
Input → State → Media
speed → energy → particle density
energy → sound activity
Audio and visuals can emerge from the same underlying process rather than being separately synchronized.
Examplary interaction archetypes
Examplary interaction archetypes
Instrument
Instrument
The body performs the media directly: pluck, bow, strike, conduct, strum, breathe.
This can be powerful when the physical action already has a temporal vocabulary.
Material
Material
The user pushes, stretches, compresses, pours, bends, or excites a virtual substance.
Sound and visuals can emerge from the material state rather than directly from the hand.
Field
Field
The hand changes an invisible region of force, attraction, resonance, turbulence, or probability.
This is particularly useful in XR because the interaction can occupy three-dimensional space.
Conductor
Conductor
The system has its own behavior.
The participant steers or influences it instead of controlling every detail.
The result can feel alive because the media has agency of its own.
Trace
Trace
Gesture leaves something behind: a sound, path, particle structure, memory, or spatial object.
The participant can later encounter the consequences of an earlier action.
Reference work
David Rokeby - Very Nervous System (1986-1990)
David Rokeby - Very Nervous System (1986-1990)
Interaction becomes an evolving encounter, not a command interface.
Very Nervous System uses cameras, image processing, computers, synthesizers, and loudspeakers to create an environment in which body movement generates sound.
Instead of recognizing fixed gestures, the system analyzes qualities of movement such as velocity, acceleration, direction, and dynamics and maps them across multiple sound parameters.
The important lesson is not “motion controls music.” Rokeby describes the central work as the changing relationship between the participant and the sounding system. The participant moves, hears a response, changes their movement, and the feedback loop continues.
"The participant gradually discovers the behavior of the system rather than memorizing a fixed gesture vocabulary."
Golan Levin + Zachary Lieberman - Messa di Voce (2003)
Golan Levin + Zachary Lieberman - Messa di Voce (2003)
In Messa di Voce, the performers’ voices are analyzed in real time using features such as pitch and spectral density, while computer vision tracks the position of their heads and bodies. The system turns different vocal qualities into animated visual forms that appear to emerge directly from the performers’ mouths.
These graphics do not disappear immediately. They become persistent virtual objects that can move, fall, collide, and be touched by the performer’s silhouette. In one scene, vocal sounds generate floating bubbles; when a falling bubble touches the performer’s body, the sound stored in that bubble is played back. The performer can therefore create a sound, see it become an object, and later use body movement to trigger it again.
"media from a temporary output into something that can be created, stored, manipulated, and replayed through bodily interaction."
Kinfolk is a real-time interactive installation created with technologies including game engines such as Unity, motion capture, and procedural animation. The participant’s movement activates and transforms digital characters, while generative rules continuously change their appearance, textures, and behavior. The result is not a fixed animation: the artwork only fully emerges through participation, and each interaction can produce different outcomes.
body movement → system state → evolving digital organism
This combination of interaction + procedural behavior + unpredictability can make a digital system feel more alive.
Sougwen Chung - Assembly Lines (2022)
Sougwen Chung - Assembly Lines (2022)
Assembly Lines is a performance and installation in which Sougwen Chung draws alongside a custom multi-robotic system. An EEG headset measures the artist’s biofeedback, which influences the robots while human and machine draw together. Contact microphones capture the physical sounds produced by brushes and robotic movement, and these sounds are transformed and distributed through an 8-ch spatial audio system surrounding the audience.
human state → biofeedback → robot behavior → physical movement → drawing + spatial sound
Interaction does not have to mean directly controlling an output.
A human action or internal state can influence an autonomous system, which then produces its own physical and audiovisual behavior.
The participant can act more like a conductor or collaborator than a controller.
This is useful when designing interactive systems that should have their own agency and evolve over time.
Rafael Lozano-Hemmer - Embodied Light Beacons (2022)
Rafael Lozano-Hemmer - Embodied Light Beacons (2022)
In Embodied Light Beacons, participants stand behind giant human-like figures made from beams of light. The system tracks the participant’s arms, legs, and head, and maps those movements onto the corresponding limbs of the enormous light figure. Small bodily gestures are therefore amplified into large beams that move across the landscape and night sky, accompanied by synchronized droning and movement sounds.
A gesture does not need to reproduce its physical scale.
small body movement → tracking → transformation → environmental-scale light and sound
In XR, a hand moving only a few centimeters does not need to move a virtual object by the same amount. Gesture can instead control scale, force, fields, environmental change, or other amplified behaviors.
The important design question is therefore not simply “How do I track this gesture?” but “What does this gesture become in the mediated world?”
Illuminate (2023, Chelsi Alise Cocking) is an interactive art installation in which the movements of a person through open space are visually augmented and brought to life in front of them in real time through custom-coded interactive visualization software. Seamlessly merging physical and digital space, Illuminate submerges us into an artificial reality in which our usually unseen paths of movement become visible– seemingly levitating in space. This aims to give us a visceral yet magical moment in which we can see, interact with, and play with our once invisible wakes of motion—pushing the boundaries of our senses and making the invisible visible. The project also explores the themes of spatial computing, bodily expression, abstraction, and choreographic interfaces.
Maestro (2024)
Maestro (2024)
Maestro is a strong contemporary example because conducting already has a meaningful gestural language: tempo, cueing, dynamics, crescendos, accents, and pauses are expressed through movement over time.
The developers found that controller weight and grip made subtle conducting motion feel less natural.
Hands made the motion lighter and more expressive.
They also had to design around tracking loss and variation rather than assuming perfect sensing.
"the expressive capacity of hands is essential to the concept."
Myungin Lee, "Deep Neural Network based Music Source Conducting System," International Computer Music Conference (ICMC), 2018
Hand Physics Lab
Hand Physics Lab
Hand Physics Lab explores direct physical relationships between hands, fingers, objects, gravity, tools, drawing, force fields, and virtual materials.
The important part is not the number of gestures.
It is that the virtual objects provide affordances: things look and behave as though they can be touched, grabbed, pushed, balanced, or manipulated.
GDC Presentation from Meta (APRIL, 2026)
Start with hands, not controllers.
Design the interaction around natural hand movement from the beginning.Choose actions that make sense for hands.
Use gestures when the movement itself is meaningful, such as conducting, grabbing, pointing, or casting.Design for imperfect tracking.
Avoid interactions that require one exact pose at one exact moment.Allow variation in gestures.
Different users perform the same gesture differently, so recognition should be tolerant.Use strong visual and audio feedback.
Because hands have no controller haptics, users need clear feedback that an action was recognized.Let affordances teach the interaction.
Objects and environments should suggest how they can be touched, moved, or manipulated, rather than relying on instructions.
Assignment #2 - Interactive Computation
Assignment #2 - Interactive Computation
Goal:
Create a data & simulation driven interactive landscape utilizing a computational algorithm, including the Boids system as a core part of it.
Use gesture-based interaction to change the behavior of the Boids system.
You may use:
MediaPipe, MagicLeap, or etc. with camera-based hand/body tracking
XR hand tracking using Quest 2, Quest 3, or Quest 3S
The interaction can affect attraction, repulsion, alignment, cohesion, separation, velocity, density, direction, or spatial organization.
The Boids do not need to represent birds or fishes. They can represent particles, stars, neurons, organisms, data, sound agents, or other abstract systems.
Submission form: PDF including the report and github repo link (1 to 2 page)
Rubric (15 points)
(3 points) Introduction + Motivation: Design an interactive system using Boids and describe the motivation
(2) Design motivation. e.g. motivated from the swarm of birds, fishes, stars, neurons, data, theory, philosophy, story... describe how the visual and behavioral design is related to the motivation
(1) Include reference (link, image)
(3 points) Methods: Describe the system implementation
(3) Design justification with a diagram (mapping between gesture - Boids/system parameters - resulting behavior)
How does the gesture change the Boids behavior?
Why is this gesture appropriate for the interaction?
How is that aligned with the motivation?
(9 points) Demo: Create the live interactive landscape and demonstrate
(7) Demonstration
Include a working Boids simulation
Use gesture-based interaction with MediaPipe or XR hand tracking
Demonstrate meaningful changes in the Boids/system behavior
The interaction should change the computational behavior, not only work as a button or on/off control
(2) Submit the github repo including a 15 second video of your work (embedded in github Readme or provide the link)
Page updated
Report abuse