FORMAT

We will offer (at least) six sessions on multimodal semantics in April-May 2026. 

TOPIC

Formal semantics and pragmatics used to be solely concerned with spoken languages. Not any more. In recent years, as part of a more general multimodal revolution in linguistics, there has been a flurry of formal work on the meaning of sign languages, gestures, and even visual animations and emojis. How has our understanding of meaning changed as a result?

Sign languages have been a primary source of insights, and have made two major contributions to the foundations of semantics. First, categories that had been developed for spoken languages (= 'speech') turned out to be illuminating for sign languages (= 'sign') as well, but occasionally in a more transparent form (this has been proposed for logical variables, for context shift, and for telicity marking, which are arguably overtly marked in sign languages). Second, sign languages combine the same kind of logical and grammatical structure as spoken languages with a much richer iconic component. This has made it essential to devise more expressive formal systems that combine a logical and a pictorial component, hence a very fruitful interaction with recent work on the formal semantics of pictures (notably by Greenberg and Abusch).
The importance of iconicity in sign languages has led comparative linguistics to adopt a motto, due to Goldin-Meadow and Brentari, according to which sign with iconicity should be compared to speech with gestures rather than to speech alone. This, in turn, led to detailed work on the formal semantics and pragmatics of different gesture types (depending on whether they co-occur, follow or fully replace spoken words). Gestures have also offered surprising insights of their own. Despite their rarity, gestures that fully replace words arguably have a proto-grammar that is sometimes reminiscent of the (incomparably more sophisticated) grammar of sign languages, hence new questions about some linguistic universals. And novel iconic gestures seem to be integrated on the fly within a sophisticated inferential typology, with a division of their meaning among assertion, presupposition, implicature, etc.
This finding raises questions about the 'generative engine' behind various semantic and pragmatic phenomena, and in particular its ability to integrate in similar ways non-standard forms such as pictures, visual animations, and emojis.

Notes: 

(i) The seminar will not presuppose any knowledge of sign language or of sign language linguistics.  

(ii) The course does not purport to offer a general survey of multimodal semantics. It is heavily tilted towards the interests of the instructor's team.  So we won't be able to do justice to all interesting works conducted by several other teams, both theoretical and experimental.