Dialogs and Sense Squares: Flexible Performer, Audience, System Interactions
Rhema Linder, Federico Burch, Fuhao Shi, and Megan Shipman
Texas A&M University
College Station, TX

Figure 1 The visualization of our system. It is shown in Viz-a-Gogo.
We incorporate the audience and performer in an interactive experience that engaged play among them in Dialogs. To foster audience participation, we incentivize interaction by rewarding crowd movement with larger visualizations. However, we attempt to balance the tension between audience and performer control of the performance space.

Our intent is to create playful dialog between the audience, performer, and technology. We use a modified system, Sense
Squares, without a performer in an art exhibition, creating a more ambient interactive installation.

Technical Components
In our Dialogs, we sense the dancer’s movement with a Wiimote, the audience or crowd with a webcam. OpenCV is used to sense the crowd and sends UDP messages to the our visualization system.

Sensing the Crowd
We use a webcam to detect the crowd movements. Pixel differences between two sequential frames are used for the density estimation. OpenCV is used to process the image data.

In detail, there are mainly two stages.

Offline Stage
Region of Interest Selection and Space Discretization
Not the whole view of

a camera is good for crowd movement analysis, such as the roof area. Thus, we provide an interaction interface to select region of interest.


analysis the density of movement of the audience, a grid of image rather than pixel is more appropriate to be a unit, and the size of grid depends on the scene. Thus, we also provide adjustment for the preferred number of grids imposed on the region of interest.

Figure 2 Offline region of interest selection and grid setting
Online Stage
Multiple Mode Density Estimation

e use the pixel difference of current frame and previous frame to represent the movement of audience, based on which densities for each grid are estimated.

Instead of
isolated grids, we use a Gaussian function to distribute the  from every particular grid to all the grids. In this way, every individual could affect a reasonable region rather than his/her grid only. Gaussian function effectively constrained this
Figure 3 The process of density estimation. Accumulated density is estimated by a Gaussian function to enable multiple modes extraction.

Sensing the Performer
The performer uses a Wiimote to direct visual elements. As she increasing the intensity of her swings, the added acceleration amplifies the sonification and her influence on agent movement. Pointing her arm up or down moves agents up or down, rolling the Wiimote to either side moves them horizontally see Figure 4. This gives her control over the direction of the movement if agents above that of the crowd. At first, the Wiimote just scattered agents. We did find that attaching the Wiimote. A useful mental model of how to control agents in this modality is to thing of the Wiimote as an airplane that has tilt and roll. Pointing downward at a 45 degree angle will move agents down and to the side. This affords a kind of banking. Turning the Wiimote upside-down, however, breaks this kind of mapping. In addition, moving to quirky affects the Wiimote’s ability to understand tilt and roll correctly, which had to be learned and practiced by the performer.
Figure 4 Wiimote control in Dialogs.

For more details, please refer to the paper.


e use the performance in both class and Viz-a-Gogo. Here is the documentation