SENSORY PAINT - Interactive surfaces

Problem: Worldwide, one in 59 children has Autism Spectrum Disorder (ASD), and 80% of them have motor problems, such as delayed or anticipated reactions and excessive or insufficient use of strength when they conduct a motor movement. Neurological Music Therapy (Nero-MT) uses live music to improve the timing and strength regulation of children with autism. However, conducting Neuro-MT sessions for children with autism is not an easy task given their motor and attention problems. These limitations could affect the interaction of children with autism with traditional musical instruments. Elastic displays are interactive surfaces with touch capabilities allowing temporary deformations. Elastic displays offer a natural and casual way to interact and provide a multisensory experience that can help children with autism interact easily with music and stay focused during therapy.

Hypothesis: We hypothesize that an elastic display will outperform traditional tools during Neuro-MT in improving the coordination of children with ASD. In this research project, we present the design, development, and evaluation of an elastic surface to support Neuro-MT, called BendableSound.

Keywords: Elastic display, depth camera, movement, ASD, children, motor coordination

BendableSound is a large-scale elastic display enabling children with autism to play sounds when practicing coordination movements when tapping, pushing or pinching a fabric. BendableSound includes musical notes arranged in an ascendant way, and a 3D animated neon-ish background of space nebulas with translucent spacebased elements, such as rockets and planets. BendableSound includes open-ended and structured activities with challenges related to rhythm and strength. BendableSound uses a 1.5 m3 PVC structure hanging a spandex fabric. Behind the fabric (inside the frame) we use a Kinect to detect users’ interactions, a short-throw projector to display the animations and speakers to play the sounds –this hardware is connected to a PC running the gameplay dynamics.

<<Population>> using the <<prototype>> protoype

Interaction measurements: BendableSound uses a Microsoft Kinect sensor and the TSPS2 library to detect users’ interactions. Users' interactions are extracted from the deformation created when a child touches the fabric. The complete trajectory of the gestures it is composed by a vector of points (as shown in the video). For each point on the graph, we stored the id, location, timestamp, depth, bounding box, and highest/closest point to the camera. We developed the multimedia interfaces of BendableSound with QML3 and JavaScript. Digital animations are displayed in a spandex fabric that acts as a digital screen. The left-image shows the graph of the gesture’s trajectory.

Evaluation methods: Twenty-two children with ASD completed 8 NMT sessions, as a part of a 2-month intervention in the Pasitos school clinic locate in Tijuana. Participants were randomly assigned to either use an elastic touch-display (experimental group) or tambourines (control group). We conducted pre- and post- assessment evaluations, including theDevelopmental Coordination Disorder Questionnaire (DCDQ) and motor assessments related to the control of strength and timing of movements.

<<Population>> using <<traditional tools>> (Control group)

<<Population>> using <<prototype>> (Experimental group)

Outcomes and results: All participants improved their coordination, according to the DCDQ scores, and exhibited better control of their movements according to the strength and timing assessments after the intervention. Participants who used the elastic touch-display scored higher on the DCDQ.

Full demo showing how <<population>> use <<prototype>> in practice

Monica Tentori, PI

Rodrigo Zalapa, Lead Student

We thank other collaborators involved in this project including <<collaborators>> and our sponsors:

Page updated

Report abuse