A vision for adaptive and multimodal nature of language

Languages can be expressed and perceived not only through speech or written text but also through visible body expressions (hands, body, and face) or visual graphics (e.g., drawings, emojis etc.). All spoken languages use gestures along with speech, and in deaf communities all aspects of language can be expressed through the visible body in sign language. Thus, expressing  language through visual as well as other modalities is an inherent nature of language. In the last years there has been growing studies and theoretical framings to understand how expressions in different modalities and representational formats can be used together to convey an integrated message at levels of speech acts, grammar, discourse and semantics in individuals  or in interactional contexts. In this talk I will present studies at these different levels and discuss different proposals for integrating how visual features of language, along with speech or in sign languages, constitute a fundamental aspect of the human language capacity, contributing to its unique flexible and adaptive nature in adults and children.

Recommended reading: https://onlinelibrary.wiley.com/doi/10.1111/tops.12728