ZAS 26 - Multimodal Semantics Mini-course
April-May 2026
LINGUAE, Institut Jean-Nicod, CNRS
Embedded Files
FORMAT
FORMAT
TOPIC
TOPIC
Formal semantics and pragmatics used to be solely concerned with spoken languages. Not any more. In recent years, as part of a more general multimodal revolution in linguistics, there has been a flurry of formal work on the meaning of sign languages, gestures, and even visual animations and emojis. How has our understanding of meaning changed as a result?
Formal semantics and pragmatics used to be solely concerned with spoken languages. Not any more. In recent years, as part of a more general multimodal revolution in linguistics, there has been a flurry of formal work on the meaning of sign languages, gestures, and even visual animations and emojis. How has our understanding of meaning changed as a result?
Sign languages have been a primary source of insights, and have made two major contributions to the foundations of semantics. First, categories that had been developed for spoken languages (= 'speech') turned out to be illuminating for sign languages (= 'sign') as well, but occasionally in a more transparent form (this has been proposed for logical variables, for context shift, and for telicity marking, which are arguably overtly marked in sign languages). Second, sign languages combine the same kind of logical and grammatical structure as spoken languages with a much richer iconic component. This has made it essential to devise more expressive formal systems that combine a logical and a pictorial component, hence a very fruitful interaction with recent work on the formal semantics of pictures (notably by Greenberg and Abusch).
The importance of iconicity in sign languages has led comparative linguistics to adopt a motto, due to Goldin-Meadow and Brentari, according to which sign with iconicity should be compared to speech with gestures rather than to speech alone. This, in turn, led to detailed work on the formal semantics and pragmatics of different gesture types (depending on whether they co-occur, follow or fully replace spoken words). Gestures have also offered surprising insights of their own. Despite their rarity, gestures that fully replace words arguably have a proto-grammar that is sometimes reminiscent of the (incomparably more sophisticated) grammar of sign languages, hence new questions about some linguistic universals. And novel iconic gestures seem to be integrated on the fly within a sophisticated inferential typology, with a division of their meaning among assertion, presupposition, implicature, etc.
This finding raises questions about the 'generative engine' behind various semantic and pragmatic phenomena, and in particular its ability to integrate in similar ways non-standard forms such as pictures, visual animations, and emojis.
Notes:
Sign languages have been a primary source of insights, and have made two major contributions to the foundations of semantics. First, categories that had been developed for spoken languages (= 'speech') turned out to be illuminating for sign languages (= 'sign') as well, but occasionally in a more transparent form (this has been proposed for logical variables, for context shift, and for telicity marking, which are arguably overtly marked in sign languages). Second, sign languages combine the same kind of logical and grammatical structure as spoken languages with a much richer iconic component. This has made it essential to devise more expressive formal systems that combine a logical and a pictorial component, hence a very fruitful interaction with recent work on the formal semantics of pictures (notably by Greenberg and Abusch).
The importance of iconicity in sign languages has led comparative linguistics to adopt a motto, due to Goldin-Meadow and Brentari, according to which sign with iconicity should be compared to speech with gestures rather than to speech alone. This, in turn, led to detailed work on the formal semantics and pragmatics of different gesture types (depending on whether they co-occur, follow or fully replace spoken words). Gestures have also offered surprising insights of their own. Despite their rarity, gestures that fully replace words arguably have a proto-grammar that is sometimes reminiscent of the (incomparably more sophisticated) grammar of sign languages, hence new questions about some linguistic universals. And novel iconic gestures seem to be integrated on the fly within a sophisticated inferential typology, with a division of their meaning among assertion, presupposition, implicature, etc.
This finding raises questions about the 'generative engine' behind various semantic and pragmatic phenomena, and in particular its ability to integrate in similar ways non-standard forms such as pictures, visual animations, and emojis.
Notes:
(i) The seminar will not presuppose any knowledge of sign language or of sign language linguistics.
(i) The seminar will not presuppose any knowledge of sign language or of sign language linguistics.
(ii) The course does not purport to offer a general survey of multimodal semantics. It is heavily tilted towards the interests of the instructor's team. So we won't be able to do justice to all the interesting work conducted by several other teams, both theoretical and experimental.
(ii) The course does not purport to offer a general survey of multimodal semantics. It is heavily tilted towards the interests of the instructor's team. So we won't be able to do justice to all the interesting work conducted by several other teams, both theoretical and experimental.
(iii) If you wish to submit homeworks / take the class for credit, please contact the instructor by email (and see below).
(iii) If you wish to submit homeworks / take the class for credit, please contact the instructor by email (and see below).
ANNOUNCEMENTS:
ANNOUNCEMENTS:
You can provide ongoing feedback about the mini-course here.
REGISTERING
REGISTERING
If you wish to be added to the mini-course's mailing list, please fill out this form. (This will help the instructor know about the audience's background and interests, and make it possible to send them additional materials by email if relevant.)
SYLLABUS
SYLLABUS
See below.
[You can go to the Zoom 2026 version of this course to get an idea of what lies ahead.]
PREREQUISITES
PREREQUISITES
At least one course in formal semantics.
READINGS AND SLIDES
READINGS AND SLIDES
If they are not linked below, they should be in this Dropbox folder.
HOMEWORKS
HOMEWORKS
If you are interested in submitting homeworks, please contact the instructor by email.
Homeworks: 1. one squib 2. one mini-term paper, details below.
Due date for the squib: Wednesday, April 22 by email (philippe.schlenker@gmail.com)
Due date for the mini-term paper: 10 days after the last class, also by email.
REFERENCES
REFERENCES
See below for details.
General surveys
Sign language
Schlenker, Lamberton and Kuhn, Sign Language Semantics
Schlenker, Visible Meaning
Some chapters of What it All Means will be made available as optional, non-technical background readings.
Additional reading:
Kathryn Davidson. Formal semantics and pragmatics in sign languages. [draft of a book manuscript] (from the author's research page)
Gestures
Ebert, Semantics of Gesture
Schlenker, The Meaning and Grammar of Pure Gestures: Theoretical Insights
For the sign-gesture connection, see also: Goldin-Meadow and Brentari 2017
SCHEDULE AND READINGS
SCHEDULE AND READINGS
(tentative - to be adapted as we go!)
Readings that are not linked below should be in this Dropbox folder.
Wednesday, April 15
Introduction + Overview of Sign Language Semantics I
Readings: Schlenker, Lamberton and Kuhn, Sign Language Semantics
Introductory alternative: Schlenker, What it All Means Chapter 2 (general audience)
Optional [and long]:
Schlenker, Visible Meaning
Monday, April 20
Overview of Sign Language Semantics II
Readings: Schlenker, Lamberton and Kuhn, Sign Language Semantics
[same as for Session 1; see above for alternatives]
Start reading: Schlenker and Lamberton 2024 'Iconological Semantics'
Wednesday, April 22
Iconological Semantics I
Readings: Schlenker and Lamberton 2024 'Iconological Semantics'
[Deadline for the mini-squib]
Monday, April 27
Iconological Semantics II; Iconic syntax. Applications to Role Shift.
Readings: Schlenker et al. 2023 'Iconic Syntax'
Wednesday, April 29
Iconological Pragmatics I: Typology of iconic enrichments
Readings: Schlenker 'Iconic Pragmatics'
Alternative [shorter, general audience]: What it All Means chapter 12
Monday, May 4
Iconological Pragmatics II: The typology of linguistic inferences in gestures.
Readings: Tieu et al. 2019 'Linguistic Inferences without Words'
Optional: Schlenker et al. 2025 'The Inferential Typology of Language: Insights from Sign Language'.
[Friday, May 15 - deadline for the mini-term paper]
Instructions for the mini-squib
Instructions for the mini-squib
Length: at most 1 single-spaced page
A squib is similar to a mini-term paper, except that (i) it is shorter, at most 1 page, (ii) it is less advanced scientifically. You may present a puzzle without a solution (and discuss extension of the puzzle and possibly of solutions for the mini-term paper). [The online journal Snippets gives good examples of what very brief linguistic analyses can be.]
You should consult with the instructor as early possible to get feedback about potential squib topics.
Instructions for the mini-term paper
Instructions for the mini-term paper
Length: at most 4 single-spaced pages
The mini-term paper is intended to discuss (i) a new empirical or formal problem, and if possible (ii) sketch one or several possible solutions. Empirical and/or formal precision are essential: the goal is to make a contribution to a very narrow problem, if possible a new one. The mini-term paper can of course build on the squib. [The online journal Snippets gives good examples of what very brief linguistic analyses can be.]
INTRODUCTION: introduce the general empirical or formal problem you will be discussing, and announce what the main finding will be.
BACKGROUND: discuss very briefly existing analyses that are relevant for the problem you are discussing (references should appear in the bibliography). In view of length restrictions, this part should be minimal; you should primarily ensure that crucial references are cited.
PROBLEM: discuss in detail the empirical or formal problem you have uncovered, for instance:
(i) a set of data for which an analysis we have discussed makes incorrect predictions,
(ii) a potential application of an analysis to significantly new examples or even data types [e.g. applying to emojis what was developed for gestures]
(iii) a new set of data that our analyses have no account for [if so, you should eventually try to state a clear generalization, i.e. a describe rule that predicts the data; sometimes this is hard!]
SOLUTION: discuss possible solutions to this problem. For instance:
if (i): how to repair or replace the theory that makes incorrect predictions;
if (ii): show in detail how the analysis applies to some of your new examples;
if (iii): if possible, propose a new theory to derive your new data/your new generalization.
You may want to state additional predictions that should be tested in future research.
Page updated
Google Sites
Report abuse