How do visual features combine?
Background
Color and shape are ubiquitous visual features in design. For example, to make a button easily distinguishable from its surroundings, we can change its color, shape, or both.
One thing we may want to know is, how do users use color and shape information to find the button?
What kind of benefit do we get when we use both color and shape features in our design, compared to using only one of them?
Question
How do color and shape information combine in human vision?
Our Strategy
Step 1: quantify how color alone affects visual cognition
This is measured by how fast people find a target object among distractor objects when the two differ in color only.
Step 2: quantify how shape alone affects visual cognition
Similarly, it is measured by how fast people find a target object among distractor objects when the two differ in shape only.
Step 3: quantify how color & shape together influence visual cognition
We developed different mathematical models for how, in theory, color and shape information might combine in human vision.
Then we measured performance of finding the target among distractors when the two differ in both color and shape.
Finally, we compared how well the different models predicted the observations.
Operationalization
How we manipulated color/shape information?
Created pairs of visual stimuli (target & distractors) that differ in
1) color, 2) shape, and 3) both dimensions
How we measured the influence of color/shape information on human's visual cognition?
Measured how long it takes for people find the target among distractors when the two differ in color / shape / both dimensions. The dependent variable is search efficiency D.
Study Preparation
Experiments were programmed in MATLAB.
Sample
200 + participants took part in the study.
Experimentation:
Visual Search
Experiment 1:
An example condition: looking for the cyan object among orange objects
Experiment 2:
An example condition: looking for the semicircle among diamonds
Experiment 3:
An example condition: looking for the cyan semicircle among orange diamonds
Analysis:
Computational Modeling
Computed the search efficiencies when only color information is available (Experiment 1) and when only shape information is available (Experiment 2).
Predicted search efficiency when both color and shape information is available and see how good it fit to the observed data in Experiment 3.
Results
Long story short, when both color and shape information are available, people find the object faster compared to when only one of them is available.
The winning model (capturing 93% of the variability in the observed data) suggests that the combined information provided by the color and shape dimensions together is simply the summation of information provided by individual dimensions. The logic of the model can be visualized as below:
A little more details about (the success of) the model's performances: among the total of 90 datapoints that were predicted, the mean error (the difference between the observed and predicted response time) is 13 milliseconds - which suggests a pretty accurate prediction!
Comparing the winning model to the other two alternative models, one thing is clear: people tend to utilize all the available information in color and shape to the full extent. People are not picking one of them to rely on, nor are they combining the two in some less efficient ways.
Implications
When both color and shape information are available, people optimally combine and use both to speed up their performance.
So better to have both color and shape information available for users when designing your products!
Delivery
This study was published on Scientific Reports, also presented as a poster at Vision Science Society 2018 Annual Meeting.
It was also presented in 2 seminar talks at Psychology Department at University of Illinois at Urbana-Champaign.
Further Exploration
What information will you use to locate the Fuji apple? Red? Round? Certain texture? Or all of them?
A follow-up study on how shape and texture information combine was conducted and presented as a poster at Vision Science Society 2020 Annual Meeting, and published on Scientific Reports.
Long story short: people utilize shape & texture information less efficiently compared to color & shape, but still it's better than having only shape or only texture dimension.
I am now working on expanding this study to objects defined by three visual properties (color, shape and texture). I am incorporating a Bayesian Probability approach in the computational modeling of the triple feature combinational search.