This project aims to investigate whether cross-modal semantic congruence is advantageous for visual search and clarify and model the audio-visual interactions that make it possible. Participants localised a target object in a naturalistic scene image. The image was presented with a sound that was congruent or incongruent to the target or with no sound, and targets were cued either visually or auditorily. A computational model was also designed to use sounds to localise target objects in scene images. Results showed that response times were faster when images were presented with a congruent sound compared to an incongruent sound, suggesting that cross-modal semantic congruence enhance the salience of the visual object and effectively guides attention. Cross-modal advantages were observed for visual but not auditory cues, suggesting that sounds may activate a visual representation indirectly through an intermediate semantic representation. The computational model failed to generalise from training to validation, possibly due to a limited dataset, highlighting how audio-visual interactions in humans may be due to learned associations.
Coming soon.