Experiments

Experiments will be run at different scales of proxemics, involving visual, auditory, and tactile perception, coupled with human action. The mappings between renderings in different modalities will be developed with an exploratory attitude, and validated through intersubjectivity, as commonly practiced in basic design and in experimental phenomenology . On a small set of selected mappings, psychophysiological experiments will be run with participants asked to produce or react to multisensory trajectories, and data collected. The production and playback of multiple trajectories allows to investigate multisensory divided attention and continuity effects. Trajectories will be produced and reproduced with different expressive qualities, which can be specific to one attribute, to a whole gestalt, or to relations between gestures, but are inherently trans-objective. Expressive qualities are reducible to invariant spatio-temporal structures of a few points, thus being transposable between modalities through phenomenological exploration (RQ1, RO1).

Experimental setting 1: Trajectories by drawing

1.  Production: The participant is briefly exposed with a trace and asked to produce a generating trajectory with a drawing gesture on a wide interactive pen display. Kinematic, kinetics and electromyographic data are collected and used to synthesize trajectories in non-visual modalities. This is repeated for several traces in a set.

2.  Reproduction: The participant is exposed to the tactile or auditory rendering of the trajectories and asked to reproduce them by drawing.

 This procedure allows to test for consistency of trajectories when moving across different modalities, possibly introducing distortions and variations in the cross-synthesis. The effect of spatial dilation/contraction and spatio-temporal subsampling can be measured through measures of consistency of drawn trajectories. The drawing gestures can be accompanied by corresponding vocal gestures that can be used to extract invariant elements for the sonic embodiment of trajectories.

Experimental setting 2: Trajectories by body movement

1.  Production: this may concern (a) the same traces of experimental setting 1 to be produced by moving a specific body part (e.g., an infinite sign produced with a hand, an elbow, the head, a knee, a foot, the trunk), (b) every-day and sport-related actions connected with the envisaged application areas (e.g., sit-to-stand, stepping, taking an object, dropping an object somewhere, throwing an object, looking for something, a volley serve, a tennis serve, kicking a ball, and so on), and (c) micro-dances, i.e., short choreographies that are performed by stressing a specific movement quality (e.g., the same choreography performed in a fluid, rigid, hesitant, aggressive way). Motion capture and electromyographic data are collected and used to synthesize trajectories in non-visual modalities.

2.  Reproduction: The participant is exposed to the tactile or auditory rendering of the trajectories and asked to reproduce them by moving either a specific part of the body or the whole body. 

The procedure allows to study whether and how production of trajectories scales from hand-drawing to other parts of the body and to the full-body (i.e., to different spatial and temporal scales). Moreover, it enables to identify possible movement primitives and to detect and measure possible supramodal qualities of trajectories with a particular reference to the envisaged application scenarios.

Experimental setting 3: Trajectories by ensembles of bodies

1.  Production: this mainly concerns a joint action where participants (either dyads or groups) collaborate in producing trajectories by means of the coordinated movement of their bodies. Trajectories may concern (a) the same traces of experimental settings 1 and 2 to be produced by the ensemble (e.g., an infinite sign produced by a dyad by waving their hands or by walking around in space), (b) every-day and sport-related actions connected with the envisaged application areas (e.g., passing an object to another participant, clapping games, two participants throwing an object to each other, a pass in soccer, and so on), (c) micro-dances to be performed by small ensembles with different movement qualities. Motion capture and (possibly) electromyographic data are collected and used to synthesize trajectories in non-visual modalities.

2.  Reproduction: The ensemble is exposed to the tactile or auditory rendering of the trajectories and asked to reproduce them by moving either a specific part of the body or the whole body.

The procedure allows to study whether and how production of trajectories scales from single participants to ensembles, i.e., to investigate the role of social interaction in production and reproduction of trajectories. Moreover, it enables to detect and measure coordination and leader-follower patterns in production and reproduction of trajectories.