Summary of Progress on Google Research Award on Material Perception

Goal 

Perception is often studied in terms of the identification and interpretation of sensory information in the environment. When we recognize an object, we combine sensory cues such as shape, color, edges, and context and make a judgment of “what it is” based on the sensory input.  However, the common emphasis on recognition undervalues the importance of the active interaction between the viewer and the world. Vision allows the viewer to make predictions about the mechanical properties of objects that have not yet been touched. When we reach to pick up a glass of milk, we have already made predictions about the weight and rigidity of the glass, as well as the fluidity of the milk.

In the field of material perception, such mechanical predictions are particularly important. The fact that glass is rigid and water is fluid is crucial in taking successful action. Until recently, the field of material perception has concentrated on the passive perception of properties like gloss, translucency, and roughness. In order to achieve a richer understanding of perception and action, we need to work on multimodal problems, such as the relationship between visual, mechanical, and haptic properties. In the following projects, we use fabrics as an example of common deformable objects to study how humans judge material properties from images and videos with an emphasis on multimodal perception and dynamical properties of materials. Our initial results identified important image cues for successful material perception in complex environment. 

Current project: visual and tactile perception of fabrics

Experiment method: tactile-visual matching

One amazing ability of human perception is to be able to predict how object behave in the new environment with unknown external forces.  For example, when we online shopping for a decorative table cloth, even if the the vendor provided videos of how the table cloth was draped over a table, we have to imagine how it would drape over our own table, which might provide a different support force. This ability of extracting intrinsic mechanical properties of object and use it to predict how it would response under unknown forces is very important for human and machine intelligence. 

Using fabrics as a model system,  we are starting to ask this question by estimating material properties of cloth from observing the motion.  We started to collect a database of RGB and RGBD videos of real fabric with corresponding measured material properties and a database of RGB videos and mesh information of synthetic fabric with corresponding material property parameters. We will design a perceptual study of how well humans are able to estimate material properties of cloth, especially stiffness and mass, from videos using a paired-comparison scaling method.  We will compare the perceptual response with the physical measurement of clothes. Finally  we will develop computational method by extracting statistics from the motion field to make inference of the material properties of fabrics. The goal of our model is to be able to predict the ground truth measurement and also perceptual responses.

Predicting material properties from actions

Even  though humans are good at estimating material properties from images, The world is dynamic. It is often helpful to observe things in action. For example, if we want to know whether a piece of cheese soft or hard, the best thing to do is to use a knife to cut it or observe others cutting it. In the previous work, we discovered some image features such as how a piece of fabric drape influence our tactile and visual matching of fabrics. Here, we want to extend this work into using videos of people manipulating fabrics as input stimuli and test whether using videos will improve accuracy between visual and tactile matching of fabrics.