International Conference on Multimodal Communication: Developing New Theories and Methods
Registration is now open until 26 May 2017
Find us on Twitter for the most recent conference news!
Osnabrück University, 9-11 June 2017
Conference organizers: Alexander Bergs and Mark Turner
The conference will run from about 1:30pm 9 June to about 6:15pm 11 June 2017, Central European Summer Time
We thank, for their generous support of the conference, Osnabrück University, Case Western Reserve University, the Anneliese-Maier Research Award program of the Alexander von Humboldt Foundation, and the Deutsche Forschungsgemeinschaft program for international conferences.
In addition to plenary talks, parallel sessions, a poster session, and a conference dinner, the conference will feature, on Saturday and Sunday mornings, several plenary workshops on methods by leading methodological experts, each presenting a specific workflow, showing how specific methods can be applied to transform a research question into finished, publishable research products.
Title: On establishing an integrated research agenda and workflow for multimodal communication. Abstract: Human communication is a core research area, intersecting with every other human endeavor. An improved understanding of the actual and effective complexities of human communication will have consequences for a broad range of fields, from politics and religion to education and business. Historically, the study of human communication has long been recognized as a central discipline, reaching back to antiquity when the study of rhetoric was the core element in education. Theoretical advances in the understanding of human communication date back millennia, to the early grammarians. The focus was mostly on written language, since the data record lent itself to systematic study. Human communication, however, has always been multimodal, and modern communication technologies have been developed to allow the full visual and auditory channels of face-to-face communication to be broadcast globally. These broadcasts can in turn be electronically captured and stored, making vast datasets of real world multimodal communication for the first time available for systematic scientific study. These new datasets present a radical new challenge: to develop a new and integrative model of the full complexity of human communication, building on existing advances in linguistics. To advance research into human multimodal communication and its role in human endeavors, Mark Turner and I founded The Distributed Little Red Hen Lab. Red Hen is designed to function as a global research commons, a practical and theoretical platform for research into multimodal communication. It provides core datasets, maintains a wide and rapidly growing network of researchers, develops an expanding suite of new computational tools, and facilitates the exchange of skills and the identification of the complementary forms of expertise required to make rapid progress. It aims to create an efficacious multilevel integrated research infrastructure and workflow along the following lines.
The presentation particularly emphasises integrating analytics and empiricism for the multimodal study of human behaviour in (select) contexts: communications and media design, architecture design, and interaction design. With the support of an additional plenary tutorial, I will highlight core results and open opportunities for the semantic interpretation of human behaviour in the backdrop of “indoor wayfinding studies” and “cognitive film studies” encompassing:
I will showcase the manner in which semantic interpretation of human behaviour, founded on AI-based methods such as in (1–4), serves as basis to externalise explicit and inferred knowledge about embodied cognitive experiences, e.g., using modalities such as diagrammatic representations, natural language, complex (dynamic) data visualisations.
Title: "Crossmodal clusters: Mapping linguistic constructions and gestural patterns via motion-capture data." Abstract: As a contribution to research into multimodal constructions (e.g., Steen & Turner 2013; Zima 2014), this talk examines how dialogue partners combine linguistic and gestural devices for spatial-geographical orientation during joint travel planning. Indicating potential destinations and routes typically involves the use of highly context-dependent indexical expressions such as closed-class items (Talmy 2000) or shifters (Jakobson 1971; Mittelberg & Waugh 2014). Our starting assumption was that in spoken German discourse the use of place names and certain deictic expressions correlates with distinct kinds of gestural shifts along the main axes (i.e., vertical, transversal, sagittal). For instance, constructions (e.g., Goldberg 2006) containing locative or directional prepositions (e.g., ‘bei’ (at) or ‘zu’ (to)) tend to co-occur with other kinds of gestural practices than constructions containing locative or directional adverbials (e.g., ‘herum’ (around) or ‘rüber’ (over)). Here we focus on the following target structures:—Prepositional constructions comprising PREP + N (‘von Norden’ (from north), ‘nach Paris’ (to Paris)) or PREP + ADV (e.g., ‘nach hier’, ‘nach da’ (to here/there)) —Adverbial constructions comprising ADVlocative + ADVdirectional (e.g., ‘da rüber’ (over there), ‘hier hin’ (to here))
First, we observed two main strategies regarding spatial orientation and gestural charting: i.) indicating places (cities, countries) through locating gestures; and ii.) tracing trajectories through routing gestures. We further hypothesized that whereas prepositional constructions (as described under a.) tend to correlate with indexical locating gestures, adverbial constructions (as described under b.) may co-occur with both locating gestures and routing gestures. The latter might contain specific directional movement information not necessarily specified in the concurrent speech; Clark 2003; Cooperrider & Núñez 2009; Fricke 2007; Haviland 2000). Using the Multimodal Speech & Kinetic Action Corpus (MuSKA), our approach combines qualitative and quantitative methods to analyze three time-aligned data streams: audio, video and motion-capture data. The sub-corpus used here contains 60 minutes of annotated naturalistic discourse data. The speech transcripts were coded for shifters and the adverbial and prepositional phrases in which they occur; the video data were coded for gestural shifts exhibiting either locating or routing functions. In three dialogues (42 min. in total), we counted 300 gesture-accompanied occurrences of locative prepositions and adverbials (130 place names; 170 PREP + ADVlocative or ADVdirectional). In a subsequent mapping process, we correlated the gestures’ spatial characteristics with co-occurring linguistic structures through integrating the three data streams and the annotated transcripts into a single multimodal database. Particular attention was paid to the gestures’ spatial attributes, e.g., the primarily evoked axis, movement extension, and location in the emergent 3D gesture spaces (Priesters & Mittelberg 2013). Analyses across speakers and dialogues were facilitated through normalizing the motion-capture data and scaling the individual gesture spaces. By aggregating the spatial parameters into overlays and heat maps, we further visualized clusters of multiple gesture instances with respect to the targeted linguistic structures. Discussing questions pertaining to frequency and conventionalization, we argue that the observed patterns of correlated verbal shifters and gestural shifts may be considered multimodally instantiated constructions (e.g., Mittelberg 2017, in press; Schoonjans et al. 2016). References:
Bio: Professor of Linguistics and Cognitive Semiotics at the Human Technology Centre (HumTec) at RWTH Aachen University. Mittelberg directs the Natural Media Lab and the Center for Sign Language and Gesture (SignGes) http://www.humtec.rwth-aachen.de/index.php?article_id=953&clang=1
4. Harald Baayen
Title: “Multimodal Communication – Multimodal Constructions?” Abstract: Language is a symbolic system, whose basic units are arbitrary and conventionalized pairings of form and meaning. In fact, in light of substantive empirical evidence, Construction Grammar advocates the view that not only words but all levels of grammatical description – from morphemes, words, and idioms to abstract phrasal patterns as well as larger discourse patterns – comprise form-meaning pairings, which are collectively referred to as constructions. Besides this, usage-based Construction Grammar approaches claim that the storage, i.e. mental entrenchment, of constructions depends on the repeated exposure to specific utterances, as well as the interaction of domain-general cognitive processes such as categorization, chunking and cross-modal association. Finally, due to the fact that utterances are complex usage-events, the information that is considered to be stored in a construction comprises not only purely linguistic (i.e. phonetic, syntactic and semantic) properties but also inferences based on the social, physical and linguistic context of an utterance (Bybee 2010). Since authentic spoken utterances are very often accompanied by gesture, it seems logical to assume that cross-modal association and chunking should also result in multimodal constructions: constructions that also contain information on gesture, facial expressions, body posture, etc. The ontological status of such multimodal constructions and their implications for cognitive theories of language have, indeed, recently become the focus of recent constructionist research (cf., e.g., Steen and Turner 2013; Zima 2014; Csienki 2015; Schoonjans, Brône and Feyaerts 2015; Pagán Cánovas and Antovic 2016). In this talk, I will discuss the status of multimodal usage-events (constructs) for the potential entrenchment of multimodal constructions and their implications for human cognition in general. After a short overview of the various types of unimodal gesture constructions (including, inter alia, emblems, iconics, deictics, and metaphorics), I will take a closer look at the prototypical differences between language and gesture as related, yet different types of semiosis (McNeill 2005, 2015). As I will show, there is some evidence for the existence of multimodal constructions (which I, however, argue mainly arise in close-knit sociolinguistic groups as complex linguistic acts of identity). Yet, due to the different modes of semiosis of language and gesture, the majority of multimodal constructs must be considered parallel realisations of independent unimodal language and gesture constructions. Far from being theoretically uninteresting, however, these latter types of complex multimodal constructs raise important issues concerning the duality of constructs in general (as fluid, emergent patterns that are partly licensed by stored, prefab templates). In the final part of my talk, I will explore this duality of constructs and outline a cognitive network account that licenses such multimodal usage-events.
State-of-the art methods for studying multimodal communication, each showing one or more complete workflow arcs, from a beginning question to a final research product.
Title: “Red Hen tools for the study of multimodal constructions.” Abstract: The Distributed Little Red Hen Lab (http://redhenlab.org) has been developing new tools for several years, with support from various agencies, including Google, which has provided two awards for Google Summers of Code, in 2015 and 2016. These tools concern search, tagging, data capture and analysis, language, audio, video, gesture, frames, and multimodal constructions. Red Hen now has several hundred thousand hours of recordings, more than 3 billion words, in a variety of languages and from a variety of countries. She ingests and processes about an additional 150 hours per day, and is expanding the number of languages held in the archive. The largest component of the Red Hen archive is called “Newsscape,” but Red Hen has several other components with a variety of content and in a variety of media. Red Hen is entirely open-source; her new tools are free to the world; they are built to apply to almost any kind of recording, from digitized text to cinema to news broadcasts to experimental data to surveillance video and more. This interactive workshop will present in technical detail some topical examples, taking a theoretical research question and showing how Red Hen tools can be applied to achieve final research results. Bio: Institute Professor and Professor of Cognitive Science, Case Western Reserve University. Co-director, the Distributed Little Red Hen Lab (http://redhenlab.org). http://markturner.org
Title: “eMAEX—analyzing temporality and audiovisual composition in film and television images.” Abstract: The workshop provides a theoretical and practical introduction to the analysis and description of audiovisual images from a film and media studies perspective (Kappelhoff & Bakels 2011; Kappelhoff 2004; Scherer et al. 2014; Bakels 2014; Bakels 2016 forthcoming; Greifenstein 2017 in preparation). The workshop introduces eMAEX – the electronically-based media analysis of expressive movement images (Kappelhoff & Bakels 2011; Scherer et al 2014). The method will be theoretically grounded and then exemplified and applied in different data types, ranging from film to television. Basic principles of how cinematic movement creates meaning will be presented and explored. The first part of the workshop presets a perspective on the overall structure of discourse and an approach to macro-segmentation of audiovisual sequences. In the second part, the micro-analytical steps regarding different forms of segmentation, description and qualification will be demonstrated and determined with regard to the medium’s use of articulatory modalities. On a more practical level, the workshop will focus on how the method can be applied using annotation tools like ELAN. The workshop allows participants to gain insights into how cinematic movement itself is the basis for the construction of narrative structures and respective experiences.
Bios: Sarah Greifenstein is a Junior Professor for Media, Culture and Communication with the focus on audiovisual media at the European University Viadrina, Frankfurt (Oder). Her main research interests are affect, expression, embodiment, and metaphoric meaning making. She is part of the research group led by Hermann Kappelhoff that developed eMAEX, a systematic approach to the analysis of audiovisual images. Jan-Hendrik Bakels is a postdoctoral researcher at Cinepoetics – Center for Advanced Film Studies at Freie Universität Berlin. His research foci are audiovisual rhythm, dynamics of affect in film viewing, audiovisual rhetorics and the methodology of film analysis. He is part of the research group led by Hermann Kappelhoff that developed eMAEX, a systematic approach to the analysis of audiovisual images. See http://www.empirische-medienaesthetik.fu-berlin.de/en/emaex-system/index.html, http://www.empirische-medienaesthetik.fu-berlin.de/en/emaex-system/emaex_kurzversion/index.html, https://euv-frankfurt-o.academia.edu/SarahGreifenstein, http://www.cinepoetics.fu-berlin.de, http://www.cinepoetics.fu-berlin.de/en/index.html.
Title: "Computational Cognitive Vision for Human-Behaviour Interpretation" Abstract: This plenary tutorial focuses on application areas where the processing and semantic interpretation of (potentially large volumes of) highly dynamic visuo-spatial imagery is central: dynamic imagery & narrativity from the viewpoint of visual perception and embodiment research; embodied cognitive vision for robotics; commonsense scene understanding etc. In the backdrop of areas as diverse as (evidence-based) architecture design, cognitive film studies, cognitive robotics, eye-tracking, the tutorial will pursue a twofold objective encompassing applications and basic methods with a particular emphasis on the visual interpretation aspect of multimodality studies. The tutorial will address AI researchers in knowledge representation, computer vision, developers of computational cognitive systems where processing of dynamic visuo-spatial imagery is involved, and educators wanting to learn about general tools for high-level logic based reasoning about image, video, point-clouds and using such tools in their teaching activities. Bio: Jakob Suchan is doctoral researcher within the Human-Centred Cognitive Assistance Lab (HCC) at the Faculty of Mathematics and Informatics, University of Bremen, Germany. His research is in the area of cognitive vision (www.cognitive-vision.org), particularly focussing on the integration of vision and AI (specifically Knowledge Representation) from the viewpoint of computational cognitive systems where integrated (embodied) perception and interaction are involved. Jakob is also a member of the DesignSpace Group (www.design-space.org).
Title: "Researching co-speech gesture in NewsScape – an integrated workflow for retrieval, annotation, and analysis." Abstract: Finding co-speech gesture in a corpus is a time-consuming process that still involves a lot of manual work. Finding such gestures aligned to abstract grammatical constructions is even more difficult in standard corpora. This plenary workshop is designed to demonstrate a full research workflow within the NewsScape/Red Hen framework from the retrieval of the grammatical structures to thinning, classification, and the analysis of the manually annotated data using state-of-the-art tools developed within the Red Hen community. Bio: Peter Uhrig is Post-Doctoral researcher at the chair of English Linguistics, FAU Erlangen-Nürnberg. He studied English and French at FAU Erlangen-Nürnberg and Lancaster University. He defended his PhD on clausal subjects in English in 2013 and started to collaborate with the Distributed Little Red Hen Lab in 2014. His main research interests are syntax, corpus linguistics, collocation ,and lexicography. His recent interest in co-speech gesture stems from his work in Construction Grammar since he hopes to be able to use co-speech gesture to find out more about the storage of grammatical constructions (i.e. which ones are more likely to be stored together and which ones are more likely to be stored separately). https://www.anglistik.phil.fau.de/staff/uhrig/
Title: Methods of Gesture Analysis—analyzing multimodality from a cognitive-linguistic perspective.
Abstract:The workshop provides a theoretical and practical introduction to the analysis and annotation of gestures from a (cognitive) linguistic perspective (Bressem, Ladewig and Müller 2013; Müller 2010; Müller, Bressem and Ladewig 2013; Müller, Ladewig and Bressem 2013). The first part of the workshop introduces the Methods of Gesture Analysis (MGA), a form-based method to systematically reconstruct the meaning of gestures. It allows for the reconstruction of fundamental properties of gestural meaning creation and determines basic principles of gestural meaning construction by distinguishing four main building blocks: 1) form, 2) sequential structure of gestures in relation to speech and other gestures, 3) local context of use, i.e., gestures’ relation to syntactic, semantic, and pragmatics aspects of speech, and 4) distribution of gestures over different contexts use. The second part of the workshop presents its implementation in an ELAN based annotation system, the Linguistic Annotation System for Gestures (LASG), providing guidelines for the annotation of gestures on a range of levels of linguistic description. As such the workshop familiarizes the participants with the a cognitive-linguistic perspective on gestures that focuses on a description of the structural and functional properties of gestures (“grammar of gesture” Müller, Bressem and Ladewig 2013; Müller 2010) and an investigation of the relation of speech and gestures in conjunction from the perspective of a “multimodal grammar” (Bressem 2014; Fricke 2012; Ladewig 2014). References:
Bio: The Berlin Gesture Center. http://www.berlingesturecenter.de/corneliamueller.html Jana Bressem is academic assistant to the chair for German Linguistics, Semiotics, and Multimodal Communication at the Technische Universität Chemnitz. Her main research interests are multimodality (speech/gesture and text/image), language and cognition and pragmatics of gesture use. Together with Cornelia Müller and the ToGoG team, she has devloped a linguistic, form-based approach to gestures. https://www.tu-chemnitz.de/phil/ifgk/germanistik/sprachwissenschaft/mitarbeiter.html#bressem,www.janabressem.de, http://www.berlingesturecenter.de/corneliamueller.html. Silva Ladewig is an Assistant Professor, currently replacing Cornelia Müller at the European University Viadrina, Frankfurt (Oder). Her main research interests are multimodality, gesture-sign interface, cognitive grammar, and embodiment. Together with Cornelia Müller and the ToGoG team, she has developed a linguistic, form-based approach to gestures. https://www.kuwi.europa-uni.de/en/lehrstuhl/sw/sw0/mitarbeiter/index.html, www.silvaladewig.de, http://www.berlingesturecenter.de/corneliamueller.html
Find us on the Sociolinguistic Events Calendar:http://baal.org.uk/slxevents.html.