Meaning in Speech, Sign and Gestures

September-October 2016 - NYU

Instructor:          Philippe Schlenker        (E-mail:  philippe.schlenker@gmail.com)


Three insights have been gained from comparative semantic research on signed and spoken languages. 
1. Logical Visibility: sometimes, elements that are covert in spoken languages are overtly realized in sign languages. A case in point concerns the formal indices that are often postulated in semantics; while these are not pronounced in spoken languages, they arguably are in sign languages: in cases of binding, an antecedent sets up a position or 'locus', which is then retrieved by the bound pronoun by way of pointing.
2. Iconicity: sign languages have iconic possibilities that only exist in very limited form in spoken languages. In fact, loci themselves have this property: they can simultaneously play the role of formal indices and of simplified pictures of what they denote. In this respect, it would seem that signed languages have richer expressive resources.
3. Gestures: due to the limitations of vocal iconicity, a proper semantic comparison should be effected between signed languages and spoken languages together with co-speech gestures – which are now investigated within formal semantics.
One salient semantic question lies in the logical status of meaning enrichments afforded by iconicity and gestures. Their typology turns out to interact in interesting ways with recent work in formal pragmatics, pertaining to the difference between at-issue, presuppositional and 'supplementary' inferences.
The seminar will be devoted to a fine-grained semantic comparison between spoken and signed languages, with a particular emphasis on the role of iconicity and co-speech gestures.


Please sign up here if you plan to attend some or all of the seminar (irrespective of whether you plan to enroll or not; this is just so that the instructor has your email address and can assess how many people will attend).


Besides active class participation, at least:
(i) 1 squib/mini-literature review + 1 additional squib or class presentation
(ii) 1 mini-term paper (to be emailed 10 days after the seminar end; ideally the mini-term paper will have been prepared by the two squibs/presentations in (i))

The squib and mini-term paper should be connected to the broad typological questions that will be addressed in the seminar (including co-speech gestures and/or iconic phenomena in spoken languages, as well as sign languages).

Please contact the instructor soon after the beginning of the term to discuss (i) and (ii).


-Background readings

• Schlenker, Philippe: to appear,  Visible Meaning: Sign Language and the Foundations of Semantics. Final version to appear as a target article (with peer commentaries) in  Theoretical Linguistics.

Shorter version available as:
Schlenker, Philippe: 2016. Logical Visibility and Iconicity in Sign Language Semantics: Theoretical Perspectives [Handbook Article]. Manuscript, Institut Jean-Nicod and New York University.

• Schlenker, Philippe: 2016, Gesture Projection and Cosuppositions. Manuscript, Institut Jean-Nicod and New York University.

-Other readings: if they are not linked below, they will be made available in this Dropbox folder.


Topics discussed will be among the following (but there might not be time for the entire list):

1. Overview: typology of iconic enrichments
2. Loci in sign language and in gestures
3. Role Shift: context shift or gestures?
4. Iconic plurals and pluractionals
5. Co-speech gestures and cosuppositions
6. Post-speech gestures and supplemenents


8 weeks in September-October 2016 - see below for sessions and readings.
–General sessions (2 hours 45 per week) are open to everyone
–Student sessions (1 hour 45 per week) are open to all students and postdocs (whether registered or not for the course).

1st session: Wednesday, September 7, 9:30am-12:15am [exceptionally, this will be a general session, and it will occupy the full 9:30am-12:15pm time slot]

Note: a guest lecture by Natasha Abner (Montclair State University) is currently being scheduled (probably towards the end of the seminar).
Main sessions (Mondays 3:30-6:15pm, 10 Washington Place, 1st floor, WP 103) 

Monday, September 12: General session, Overview - continued

Readings: continue with the background readings + Abner et al. 2015: Gestures for linguists

Additional readings (optional):
Kuhn on sign language anaphora [survey]
Goldin-Meadow and Brentari 2015 BBS

Monday, September 19:  General session, Overview - continued
Optional: Kuhn and Aristodemo 2016 


Monday, September 26:  Guest speaker: Gabe Greenberg (UCLA) on iconic semantics

Title: The Structure of Visual Content

Abstract: This will be a workshop-style discussion of some new ideas in the semantic analysis of visual representation.   I’ll focus on the use of "feature maps" to characterize a wide range of visual content, with an eye towards specifying precise accuracy conditions.  The primary application will be static pictorial images (such as drawings or photographs), but we’ll also consider extensions to dynamic displays such as comics, film, and iconicity in gesture and sign language.  As time permits, we’ll discuss a range of semi-technical issues that arise for feature map semantics, including the representation of angle, depth, category, quantification, transparency, relations, and temporal duration. 

Reading: Greenberg and Giardino, Varieties of Iconicity

Monday, October 3:  Co- and post-speech gestures. Agreement with English gestural verbs

Readings Gestural Agreement

Monday, October 10:  no class [NYU holiday]

Monday, October 17: Agreement with English gestural verbs - initial experimental results on gesture projection

Readings Gestural Agreement

Monday, October 24: last general session
Discussion sessions (Wednesdays, 1 hour 45 minutes:  10:30am-12:15pm, 10 Washing Place, 4th floor, WP 408) 

Note: Wednesday, September 7 (1st day of the seminar) will be a main session and will occupy the FULL 9:30-12:15pm time slot.

Wednesday, September 7: 2-hour 45 minute session, General Introduction

Readings: start with the background readings + Abner et al. 2015: Gestures for linguists

Additional readings (optional):
Goldin-Meadow and Brentari 2015 BBS

Wednesday, September 14:  10:30am: Guest speaker: Matthew Stone (Rutgers) on gesture semantics

Title: Interpreting Multimodal Communicative Action

Abstract: I give a short tutorial overview of the nonverbal actions that
speakers use to enrich the meanings of their contributions to spoken
face-to-face conversation.  We will see the ways speakers not only
refer to real space but construct virtual and abstract spaces to
locate events; the ways they indicate geometry, represent objects and
imitate actions; and the consistent patterns that link successive
gestures into extended depictions.  Time permitting, we will also
explore the analogous ways speakers can recruit physical props and
practical activity in the service of communication.  Finally, we will
look at some of the formal devices that have been used to model the
compositional connections between language and gesture, including
generalized conjunction, function-argument dependencies and the
reconstruction of implicit inferential relationships.  A key theme
throughout the presentation will be the role of discourse coherence in
organizing content across modalities into an integrated whole.

ReadingLascarides & Stone, Discourse coherence and gesture interpretation, 2009

Lascarides & Stone, A formal semantic analysis of gesture, 2009

Wednesday, September 21:  10am: Masha Esipova: Alternatives matter: contrastive focus and presuppositions in standard triggers and co-speech gestures [slides]

Wednesday, September 28:   10am: Insa Lawler  (University of Duisburg-Essen) on gesture semantics

 A formal semantics for iconic gestures
Abstract: Face-to-face communication is often accompanied by iconic gestures. Intuitively, such co-speech gestures have meaning and contribute to the communicated content. But what is a gesture's meaning and how is it determined? In the talk, I first present the typology based approach to a formal semantics for iconic gestures, illustrated by examples from the Bielefeld Speech and Gesture Alignment corpus (SaGA). Then I point out that the common practice of annotating gestures as drawing geometrical shapes, such as circles, obscures one fundamental relation between gesture and speech: The interpretation of an iconic gesture's meaning is dependent on its accompanying speech to some degree. Humans never gesticulate perfect geometrical shapes. Whether a helix-like gesture is interpreted as conveying the meaning CIRCULAR or SPIRAL depends on the co-occurring speech part (e.g., "window" vs. "spiral staircase"). The dependence claim is further elaborated on employing data from SaGA and motion capturing data. Then I present an ongoing empirical study in which we examine the dependence claim in more detail. I end by outlining some possibilities for how to formally analyze the tackled speech-dependence of gesture interpretation.

The work presented is joint work with Hannes Rieser, Florian Hahn and Thies Pfeiffer, and based on collaborations with Stefan Kopp and Kirsten Bergmann in the CRC "Alignment in Communication".

Optional background reading:
Lücking, Bergmann, Hahn, Kopp, Rieser (2013). Data-based analysis of speech and gesture: the Bielefeld Speech and Gesture Alignment corpus (SaGA) and its applications. Journal on Multimodal User Interfaces, 7(1-2): 5-18. https://pub.uni-bielefeld.de/download/2522299/2534574
Pfeiffer, Hofmann, Hahn, Rieser, Lawler (2013). Gesture semantics reconstruction based on motion capturing and complex event processing: a circular shape example. Proceedings of the Special Interest Group on Discourse and Dialog (SIGDIAL) 2013 Conference. Eskenazi, Strube, Di Eugenio, Williams (Ed.); Association for Computational Linguistics: 270-279.http://www.sigdial.org/workshops/conference14/proceedings/pdf/SIGDIAL41.pdf

Squib due [by email, before class; in pdf or doc]

Wednesday, October 5:  no class [to be replaced]

Wednesday, October 12: 10am Guest speaker: Natasha Abner (Montclair State University):  

Language Emergence and the Gesture-(Sign) Language Continuum

 Abstract: Structural patterns in homesign provide unique insight into the potentially foundational properties of human language. The implications of evidence from homesign, however, can only be understood in light of the observation that homesign is not truly created de novo. Rather, homesigners do have access to the co-speech gesture of the ambient speaking community and this accessible system may influence the development of their idiosyncratic homesign system. Thus, in order to understand homesign and the evolution from homesign to ‘conventionalized’ sign language, it is necessary to understand the properties of co-speech gesture and the structural similarities between sign and gesture. In this talk, I will focus on these two related continua of language development, homesign-sign language and gesture-sign language, and the ways that they suggest we critically (re-)evaluate the relationships between signed and spoken language, sign language and gesture, and gesture and language. In doing so, I will highlight recent research on the expression of number (plurality, pluractionality) and event meaning (transitivity, telicity) in gesture, homesign, and sign language. 

Reading: Goldin-Meadow 2012

Wednesday, October 19: 10am: Daniel Barry: 'Variable-based features in ASL'

Thursday, October 20: 10am-11:45am-4th floor lounge: optional make-up session

Optional readings: Kuhn 2015 'ASLI Loci: variables or features?'
Schlenker 2016 'Featural variables
(both in the 'Signs 1' folder)

Wednesday, October 26: 10am: Haoze Li on metalinguistic focus

Reading: Artstein 2014

Mini-term paper due: Monday, November 7th