NYU16

Meaning in Speech, Sign and Gestures

Philippe Schlenker

(LINGUAE, Institut Jean-Nicod, CNRS; New York University)

September-October 2016 - NYU

Instructor: Philippe Schlenker (E-mail: philippe.schlenker@gmail.com)

Topic

Three insights have been gained from comparative semantic research on signed and spoken languages.

1. Logical Visibility: sometimes, elements that are covert in spoken languages are overtly realized in sign languages. A case in point concerns the formal indices that are often postulated in semantics; while these are not pronounced in spoken languages, they arguably are in sign languages: in cases of binding, an antecedent sets up a position or 'locus', which is then retrieved by the bound pronoun by way of pointing.

2. Iconicity: sign languages have iconic possibilities that only exist in very limited form in spoken languages. In fact, loci themselves have this property: they can simultaneously play the role of formal indices and of simplified pictures of what they denote. In this respect, it would seem that signed languages have richer expressive resources.

3. Gestures: due to the limitations of vocal iconicity, a proper semantic comparison should be effected between signed languages and spoken languages together with co-speech gestures – which are now investigated within formal semantics.

One salient semantic question lies in the logical status of meaning enrichments afforded by iconicity and gestures. Their typology turns out to interact in interesting ways with recent work in formal pragmatics, pertaining to the difference between at-issue, presuppositional and 'supplementary' inferences.

The seminar will be devoted to a fine-grained semantic comparison between spoken and signed languages, with a particular emphasis on the role of iconicity and co-speech gestures.

Important

Please sign up here if you plan to attend some or all of the seminar (irrespective of whether you plan to enroll or not; this is just so that the instructor has your email address and can assess how many people will attend).

Requirements

Besides active class participation, at least:

(i) 1 squib/mini-literature review + 1 additional squib or class presentation

(ii) 1 mini-term paper (to be emailed 10 days after the seminar end; ideally the mini-term paper will have been prepared by the two squibs/presentations in (i))

The squib and mini-term paper should be connected to the broad typological questions that will be addressed in the seminar (including co-speech gestures and/or iconic phenomena in spoken languages, as well as sign languages).

Please contact the instructor soon after the beginning of the term to discuss (i) and (ii).

Readings

-Background readings

• Schlenker, Philippe: to appear, Visible Meaning: Sign Language and the Foundations of Semantics. Final version to appear as a target article (with peer commentaries) in Theoretical Linguistics.

Shorter version available as:

Schlenker, Philippe: 2016. Logical Visibility and Iconicity in Sign Language Semantics: Theoretical Perspectives [Handbook Article]. Manuscript, Institut Jean-Nicod and New York University.

• Schlenker, Philippe: 2016, Gesture Projection and Cosuppositions. Manuscript, Institut Jean-Nicod and New York University.

-Other readings: if they are not linked below, they will be made available in this Dropbox folder.

Topics

Topics discussed will be among the following (but there might not be time for the entire list):

1. Overview: typology of iconic enrichments

2. Loci in sign language and in gestures

3. Role Shift: context shift or gestures?

4. Iconic plurals and pluractionals

5. Co-speech gestures and cosuppositions

6. Post-speech gestures and supplemenents

Schedule

8 weeks in September-October 2016 - see below for sessions and readings.

–General sessions (2 hours 45 per week) are open to everyone

–Student sessions (1 hour 45 per week) are open to all students and postdocs (whether registered or not for the course).

1st session: Wednesday, September 7, 9:30am-12:15am [exceptionally, this will be a general session, and it will occupy the full 9:30am-12:15pm time slot]

Note: a guest lecture by Natasha Abner (Montclair State University) is currently being scheduled (probably towards the end of the seminar).

Monday, October 17: Agreement with English gestural verbs - initial experimental results on gesture projection

Readings: Gestural Agreement

Monday, October 10: no class [NYU holiday]

Monday, October 3: Co- and post-speech gestures. Agreement with English gestural verbs

Monday, September 19: General session, Overview - continued

Readings: Gesture Projection and Cosuppositions.

Optional: Kuhn and Aristodemo 2016 [draft]

Monday, September 26: Guest speaker: Gabe Greenberg (UCLA) on iconic semantics

Title: The Structure of Visual Content

Abstract: This will be a workshop-style discussion of some new ideas in the semantic analysis of visual representation. I’ll focus on the use of "feature maps" to characterize a wide range of visual content, with an eye towards specifying precise accuracy conditions. The primary application will be static pictorial images (such as drawings or photographs), but we’ll also consider extensions to dynamic displays such as comics, film, and iconicity in gesture and sign language. As time permits, we’ll discuss a range of semi-technical issues that arise for feature map semantics, including the representation of angle, depth, category, quantification, transparency, relations, and temporal duration.

Reading: Greenberg and Giardino, Varieties of Iconicity

Readings: continue with the background readings + Abner et al. 2015: Gestures for linguists

Additional readings (optional):

Kuhn on sign language anaphora [survey]

Goldin-Meadow and Brentari 2015 BBS

Schlenker 2015

Monday, September 12: General session, Overview - continued

Main sessions (Mondays 3:30-6:15pm, 10 Washington Place, 1st floor, WP 103)

Discussion sessions (Wednesdays, 1 hour 45 minutes: 10:30am-12:15pm, 10 Washing Place, 4th floor, WP 408)

Note: Wednesday, September 7 (1st day of the seminar) will be a main session and will occupy the FULL 9:30-12:15pm time slot.

Wednesday, September 7: 2-hour 45 minute session, General Introduction

Readings: start with the background readings + Abner et al. 2015: Gestures for linguists

Additional readings (optional):

Goldin-Meadow and Brentari 2015 BBS

Schlenker 2015

Wednesday, September 14: 10:30am: Guest speaker: Matthew Stone (Rutgers) on gesture semantics

Title: Interpreting Multimodal Communicative Action

Abstract: I give a short tutorial overview of the nonverbal actions that

speakers use to enrich the meanings of their contributions to spoken

face-to-face conversation. We will see the ways speakers not only

refer to real space but construct virtual and abstract spaces to

locate events; the ways they indicate geometry, represent objects and

imitate actions; and the consistent patterns that link successive

gestures into extended depictions. Time permitting, we will also

explore the analogous ways speakers can recruit physical props and

practical activity in the service of communication. Finally, we will

look at some of the formal devices that have been used to model the

compositional connections between language and gesture, including

generalized conjunction, function-argument dependencies and the

reconstruction of implicit inferential relationships. A key theme

throughout the presentation will be the role of discourse coherence in

organizing content across modalities into an integrated whole.

Reading: Lascarides & Stone, Discourse coherence and gesture interpretation, 2009

Optional: Lascarides & Stone, A formal semantic analysis of gesture, 2009

Note: Jerry Hobbs on Granularity

Wednesday, September 21: 10am: Masha Esipova: Alternatives matter: contrastive focus and presuppositions in standard triggers and co-speech gestures [slides]

Wednesday, September 28: 10am: Insa Lawler (University of Duisburg-Essen) on gesture semantics

Title: A formal semantics for iconic gestures

Abstract: Face-to-face communication is often accompanied by iconic gestures. Intuitively, such co-speech gestures have meaning and contribute to the communicated content. But what is a gesture's meaning and how is it determined? In the talk, I first present the typology based approach to a formal semantics for iconic gestures, illustrated by examples from the Bielefeld Speech and Gesture Alignment corpus (SaGA). Then I point out that the common practice of annotating gestures as drawing geometrical shapes, such as circles, obscures one fundamental relation between gesture and speech: The interpretation of an iconic gesture's meaning is dependent on its accompanying speech to some degree. Humans never gesticulate perfect geometrical shapes. Whether a helix-like gesture is interpreted as conveying the meaning CIRCULAR or SPIRAL depends on the co-occurring speech part (e.g., "window" vs. "spiral staircase"). The dependence claim is further elaborated on employing data from SaGA and motion capturing data. Then I present an ongoing empirical study in which we examine the dependence claim in more detail. I end by outlining some possibilities for how to formally analyze the tackled speech-dependence of gesture interpretation.

The work presented is joint work with Hannes Rieser, Florian Hahn and Thies Pfeiffer, and based on collaborations with Stefan Kopp and Kirsten Bergmann in the CRC "Alignment in Communication".

Optional background reading:

Lücking, Bergmann, Hahn, Kopp, Rieser (2013). Data-based analysis of speech and gesture: the Bielefeld Speech and Gesture Alignment corpus (SaGA) and its applications. Journal on Multimodal User Interfaces, 7(1-2): 5-18. https://pub.uni-bielefeld.de/download/2522299/2534574

Pfeiffer, Hofmann, Hahn, Rieser, Lawler (2013). Gesture semantics reconstruction based on motion capturing and complex event processing: a circular shape example. Proceedings of the Special Interest Group on Discourse and Dialog (SIGDIAL) 2013 Conference. Eskenazi, Strube, Di Eugenio, Williams (Ed.); Association for Computational Linguistics: 270-279.http://www.sigdial.org/workshops/conference14/proceedings/pdf/SIGDIAL41.pdf

Squib due [by email, before class; in pdf or doc]

Wednesday, October 5: no class [to be replaced]

Wednesday, October 12: 10am Guest speaker: Natasha Abner (Montclair State University): Language Emergence and the Gesture-(Sign) Language Continuum

Abstract: Structural patterns in homesign provide unique insight into the potentially foundational properties of human language. The implications of evidence from homesign, however, can only be understood in light of the observation that homesign is not truly created de novo. Rather, homesigners do have access to the co-speech gesture of the ambient speaking community and this accessible system may influence the development of their idiosyncratic homesign system. Thus, in order to understand homesign and the evolution from homesign to ‘conventionalized’ sign language, it is necessary to understand the properties of co-speech gesture and the structural similarities between sign and gesture. In this talk, I will focus on these two related continua of language development, homesign-sign language and gesture-sign language, and the ways that they suggest we critically (re-)evaluate the relationships between signed and spoken language, sign language and gesture, and gesture and language. In doing so, I will highlight recent research on the expression of number (plurality, pluractionality) and event meaning (transitivity, telicity) in gesture, homesign, and sign language.

Reading: Goldin-Meadow 2012

Wednesday, October 19: 10am: Daniel Barry: 'Variable-based features in ASL'

Thursday, October 20: 10am-11:45am-4th floor lounge: optional make-up session

Optional readings: Kuhn 2015 'ASLI Loci: variables or features?'

Schlenker 2016 'Featural variables

(both in the 'Signs 1' folder)

Wednesday, October 26: 10am: Haoze Li on metalinguistic focus

Reading: Artstein 2014

Mini-term paper due: Monday, November 7th

Monday, October 24: last general session

Jeremy Kuhn's NASSLI course on sign language semantics

Hanne Rieser's slides on 'speech-gestures interfaces' (suggested by Insa Lawler)

Instructions for the mini-squib(s)

Choose between A, B, and C. In either case, please consult with the instructor ASAP. Registered students should write 1 squib and give a class presentation or write an additional squib.

Note: Your squib can be quite short. Ideally, it should present one very clear argument or empirical problem.

A. Write a brief review of an article that concerns one of the topics that will be discussed in the seminar (gestures, sign language, or spoken language).

Consult with the instructor on the choice of the paper. Be sure to specify in your review:

(i) the empirical problem that is addressed, and the generalizations that are discussed.

(ii) what the main formal proposal is

(iii) how the formal analysis is applied to selected examples.

(iv) what some strengths and weaknesses of the proposal are.

B. Write a squib related to one of the topics that will be discussed in the seminar (gestures, sign language, or spoken language). The squib should be very focused, and it should:

(i) define a clear problem, which could be:

(a) an empirical problem [interesting patterns of projection for connectives we haven’t discussed; empirical problems for the analyses we have discussed, etc.].

(b) a formal problem that arises in some of the theories we have discussed.

(ii) give a precise analysis of the relevant data and formalisms

(iii) discuss one or several possible solutions.

C. Give a class presentation, to be discussed with the instructor

Potential topics for student presentations

–Spoken language

• Raised eyebrows in spoken language

Dohen and others on raised eyebrows in spoken language

• Co-nominal pointing in spoken language

Anvari on co-nominal pointing (MA thesis, Paris, 2016)

• Gestures as supplements

Ebert on gestures as supplements [some slides]

• Ideophones in spoken language,

e.g. Henderson on iconic pluractionals.

–Sign language

• Pluractionals in spoken vs. sign language

Kuhn on dependent plurals: [paper] [dissertation]

Kuhn and Aristodemo. "Pluractionality, iconicity, and scope in French Sign Language." [draft]

• Functional reference in sign language

Kuhn. "Functional reference in American Sign Language." [draft]

Additional references

• Classics

Klima, Edward and Bellugi, Ursula: 1979, The Signs of Language

• Background

Padden, C.: 2011, Sign Language Geography

List of Sign Languages [Wikipedia]

• Pronouns in sign language

Lillo-Martin, Diane and Meier, Richard: 2011. On the linguistic status of 'agreement' in sign language. Theoretical Linguistics.

Sandler, Wendy and Lillo-Martin, Diane: 2006, Pronouns. From Sign Language and Linguistic Universals.

• Temporal and Modal Anaphora

Schlenker, Philippe: to appear, Temporal and Modal Anaphora in Sign Language (ASL). To appear in Natural Language & Linguistic Theory

Zucchi, Sandro: 2009, Along the time line Tense and time adverbs in Italian Sign Language, NALS.

A Classic

Partee, Barbara: 1973, Some Structural Analogies Between Tense and Pronouns in English. Journal of Philosophy 70: 601-609

More on intensional semantics

Kai von Fintel and Irene Heim's Notes on Intensional Semantics [pdf]

• Donkey Anaphora

–Donkey anaphora in sign language

Schlenker, Philippe: "Quantifiers and Variables: Insights from Sign Language (ASL and LSF)". Accepted for publication in Partee, B.H., Glanzberg, M., & Skilters, J. (eds), Formal Semantics and Pragmatics: Discourse, Context, and Models. The Baltic International Yearbook of Cognition, Logic and Communication, Vol. 6, 2011. [non-technical discussion of 'donkey' anaphora and sign language]

Schlenker, Philippe: 2012, "Donkey Anaphora: the View from Sign Language (ASL and LSF)". To appear in Linguistics & Philosophy [technical discussion of 'donkey' anaphora and sign language](non-final version; last modified: February 2011)

–Dynamic semantics vs. E-type approaches

More on dynamic semantics

Barbara Partee's Lectures on Semantics and Anaphora, Moscow, 2008 (3 lectures on dynamic semantics)

Advanced:

Heim, Irene: 1982, File Change Semantics and the Familiarity Theory of Definitiness

Elbourne, Paul: 2005, Situations and Individuals [pdf]

Dekker, Paul: 2004 Cases, Adverbs, Situations and Events, pages 1-5

Heim, Irene: 1990, E-type Pronouns and Donkey Anaphora. Linguistics and Philosophy 13: 137-177

Very Advanced:

Nouwen, Rick (2003) Plural Pronominal Anaphora in Context: Dynamic Aspects of Quantification, PhD-thesis, UiL-OTS, Utrecht University, No. 84 of the LOT dissertation-series. [pdf]

• Iconic Variables

Schlenker, Philippe: 2011, Iconic Agreement Theoretical Linguistics.

Schlenker, Lamberton and Santoro: 2012, Iconic Variables , Linguistics & Philosophy

Emmorey and Herzig: Gradient properties of ASL classifiers

• Role Shift

Emmorey and Reilly: 1998, The Development of Quotation and Reported Action: Conveying Perspective in ASL

Quer, J. and Frigola, S. The workings of indexicals in role shift structures in Catalan Sign Language (LSC)

Pyers and Senghas: 2007, Reported action in Nicaraguan and American Sign Languages: Emerging versus established systems

Schlenker, P.: to appear. Indexicality and De Se Reports" [Survey Article]. To appear in the Handbook of Semantics edited by von Heusinger, Maienborn and Portner, Mouton de Gruyter

• Ronnie Wilbur on Verbal Classes

[1. is introductory; 2. builds on 1.]

1. Wilbur, R. B. (2003). Representations of telicity in ASL. CLS 39, 354-368.

2. Wilbur, R. B. (2008). Complex predicates involving events, time and aspect: Is this why sign languages look so similar? In J. Quer (ed.), Signs of the time: Selected papers from TISLR 2004, pp. 217-250. Hamburg: Signum Press.

3. Wilbur, R. B., Malaia, E., Shay, R. A. (In press). Degree modification and intensification in ASL adjectives. 18th Amsterdam Colloquium. Springer FoLLI Publications on Logic, Language and Information.

4. Wilbur, R. B. (2009). Productive reduplication in ASL, a fundamentally monosyllabic language. In M. Kenstowicz (ed.), Data and Theory: Papers in Phonology in Celebration of Charles W. Kisseberth, a special issue of Language Sciences 31: 325-342.

• Semantics of pictures

Greenberg, Gabriel: Pictorial Semantics. Ms., UCLA. [pdf]

• Semantics of gestures

G. Giorgolo, A Formal Semantics for Iconic Spatial Gestures, in M. Aloni, B. Harald, T. de Jager and K. Schulz 'Logic, Language and Meaning, Springer', p. 305-314, 2010. [pdf]

Lascarides, A. and M. Stone [2009] Discourse Coherence and Gesture Interpretation, Gesture, 9(2), pp147--180, John Benjamins Publishing Company.

• Emergence of New Sign Languages

Ann Senghas's Homepage

Nicaraguan Sign Language

Coppola and Senghas: to appear, The emergence of deixis in Nicaraguan signing [pdf]

Senghas and Coppola: 2001, How Nicaraguan Sign Language Acquired a Spatial Grammar [pdf]

Historical Links

Note: for some of the links ['subscription'], you'll need to connect from NYU or use a proxy server so as to have access to institutional subscriptions.

• The French Deaf community before Abbé de l'Epée

A text by Deaf historian Yann Cantin [in French]

Pierre Desloges, Observations of a Deaf-Mute on an elementary course of instruction of the Deaf-Mute (1779) [in French]

• Abbé de l'Epée

Berthier, Ferdinand: L'Abbé de l'Epée [in French]

Abbé de l'Epée, The Art of Teaching the Deaf-Mute from Birth to Speak (1820) [in French]

• Abbé Sicard

Berthier, Ferdinand: L'Abbé Sicard [in French]

• Laurent Clerc

Short biography (handspeak.com)

Short biography (Encyclopedia of American Disability History)

Laurent Clerc's Address 'Before the Governor and Both Houses of the Legislature', Connecticut, 1818