Summary of Progress on Google Research Award on Material Perception
Goal
Perception is often studied in terms of the identification and interpretation of sensory information in the environment. When we recognize an object, we combine sensory cues such as shape, color, edges, and context and make a judgment of “what it is” based on the sensory input. However, the common emphasis on recognition undervalues the importance of the active interaction between the viewer and the world. Vision allows the viewer to make predictions about the mechanical properties of objects that have not yet been touched. When we reach to pick up a glass of milk, we have already made predictions about the weight and rigidity of the glass, as well as the fluidity of the milk.
In the field of material perception, such mechanical predictions are particularly important. The fact that glass is rigid and water is fluid is crucial in taking successful action. Until recently, the field of material perception has concentrated on the passive perception of properties like gloss, translucency, and roughness. In order to achieve a richer understanding of perception and action, we need to work on multimodal problems, such as the relationship between visual, mechanical, and haptic properties. In the following projects, we use fabrics as an example of common deformable objects to study how humans judge material properties from images and videos with an emphasis on multimodal perception and dynamical properties of materials. Our initial results identified important image cues for successful material perception in complex environment.
Current project: visual and tactile perception of fabrics
Bei Xiao and Edward Adelson
Humans are good at predicting how objects feel just from looking. This unique ability resulted from rich experiences in daily life through interacting with objects from both visual and tactile senses. For example, during online shopping, we use material perception to gauge how clothes would feel against our skin without being able to touch them. Almost nothing is known on what image cues are important for visual prediction of tactile and mechanical properties of objects. Studies in material perception historically have focused on measuring visual of optical properties such as gloss and roughness. But people in real life rarely verbally judge glossiness and roughness. In stead, they judge material properties in order to plan actions. In this regard, extracting mechanical and tactile properties from visual input is very useful.
In this study, we used a cross-model paired-comparison method to measure matching accuracy between visual and tactile perception. We manipulate photographs of fabrics and measure observer's visual and matching of the fabrics. Our goals are to find image cues that are important for visual prediction of tactile properties.
A. B. C. D
Different style of photographs reveals different aspects of material properties of the same fabric. A. Optical property. B. Color & Texture. C. Optical & Mechanical Property. D. Optical, Tactile and Mechanical property.
Experiment method: tactile-visual matching
During each trial, observer is asked to arrange two pieces of fabrics using their hands (without looking) inside a box so that they matched photographs of the same pairs of fabrics displayed on a monitor. Observers have 12 seconds to look at the stimuli.
Stimuli: Fabrics are made into three kinds of folds
Stimuli are photographs of 60 commonly used apparel fabrics. Fabrics are made into three different folding conditions: A. Flat fabric, where the fabric is mounted onto a flat foam board and the texture details can be seen in high-resoluation images which is shown in "Zoomed in" region. B. Draped fabric, where a fabric is draped over a bumpy object . C. Hung Fabric, where a piece of fabric is hung up from two corners.
Stimuli: removing color.
Grayscale images of the same fabrics are used in the experiment.
Results: Effect of folds and color on matching accuracy
Overall, observers are good at matching what they see and what they feel. There is a significant effect of folding shape on matching accuracy. Images of Draped fabric and Hung fabric achieved higher accuracy than images of flat fabric. We also found significant effect on color. Removing color from the images significantly reduced accuracy for Draped and Hung fabrics but not for flat folded fabrics.
Color bars represent color image conditions; grayscale bars represent grayscale image conditions. Green bar represent flat fabric condition, blue bar represents draped fabric condition, and red bar represents hung fabric condition.
Future Projects
Estimating cloth properties from videos
in collaboration with Katie Bouman and Bill Freeman
An RGBD data base of cloth videos of various dynamics are collected for both perceptual and computational study.
In the videos, cloth are undergoing external forces such as being blown by different wind forces.
A large number of fabric samples of a variety of material properties will be included.
RGB video Depth Information Light and bendable fabric Thick and stiff fabric
One amazing ability of human perception is to be able to predict how object behave in the new environment with unknown external forces. For example, when we online shopping for a decorative table cloth, even if the the vendor provided videos of how the table cloth was draped over a table, we have to imagine how it would drape over our own table, which might provide a different support force. This ability of extracting intrinsic mechanical properties of object and use it to predict how it would response under unknown forces is very important for human and machine intelligence.
Using fabrics as a model system, we are starting to ask this question by estimating material properties of cloth from observing the motion. We started to collect a database of RGB and RGBD videos of real fabric with corresponding measured material properties and a database of RGB videos and mesh information of synthetic fabric with corresponding material property parameters. We will design a perceptual study of how well humans are able to estimate material properties of cloth, especially stiffness and mass, from videos using a paired-comparison scaling method. We will compare the perceptual response with the physical measurement of clothes. Finally we will develop computational method by extracting statistics from the motion field to make inference of the material properties of fabrics. The goal of our model is to be able to predict the ground truth measurement and also perceptual responses.
Predicting material properties from actions
Even though humans are good at estimating material properties from images, The world is dynamic. It is often helpful to observe things in action. For example, if we want to know whether a piece of cheese soft or hard, the best thing to do is to use a knife to cut it or observe others cutting it. In the previous work, we discovered some image features such as how a piece of fabric drape influence our tactile and visual matching of fabrics. Here, we want to extend this work into using videos of people manipulating fabrics as input stimuli and test whether using videos will improve accuracy between visual and tactile matching of fabrics.