My paper will focus on possibilities and challenges in combining two popular contemporary tools in multimodal video analysis and biometric research. ELAN is a freeware and open source multilayer annotation tool provided by the Max Planck Institute for Psycholinguistics, Nijmegen. iMotions with its Affdex Affectiva integrated facial expression analysis engine (cf. Affectiva Blog 2020) is a commercial biometric research platform that has become a state of the art tool in all kinds of behavioural sciences, including market research, media reception analysis and emotion psychology.

While both platforms allow for basic multimodal integration of verbal and visual data as well as interpretative annotations along a timeline, specific technical constraints and affordances of both platforms with regard to specific challenges and requirements for a semiotically informed analysis will require the researchers to choose one of the platforms over the other as their central integrative annotation platform. I will seek to introduce both platforms and their functionality and provide insights into my own practical considerations as to how to choose, combine and integrate both tools.

My data is derived from German and English language talkshow interactions. English translations/subtitles will be provided. Particular attention is given to the systematic description of emotionalizing interplay of face-threatening verbal and nonverbal behaviour such as contradictions and threats and their sequential integration into and disruptive effect within multimodal interaction chains. Another aspect of my talk will be challenges and possibilities in integrating biometric and hermeneutic to interaction and behavioural analysis as discusses previously in Rothenhöfer (2021a) and Rothenhöfer (2021b) .

Affectiva Blog (2020): The Emotion behind your Conversation: Transforming the Online Qual Experience. Last accessed: 4 August 2021

(https://blog.affectiva.com/the-emotion-behind-your-online-conversation-transforming-the-qual-experience)

ELAN The Language Archive, Max Planck Institute for Psycholinguistics, Nijmegen/Netherlands. (https://archive.mpi.nl/tla/elan)

iMotions Research Software, iMotions A/S, Copenhagen/Denmark. (https://imotions.com)

Rothenhöfer, Andreas (2021a, in print): Multimodaler (Selbst-)Widerspruch. in: Julia Nintemann & Cornelia Stroh (eds.), Über Widesprüche sprechen: Linguistische Beiträge zu Contradiction Studies. Wiesbaden: Springer.

Rothenhöfer, Andreas (2021b): "I can't see why you're laughing": Multimodal Analysis of Emotionalized Political Debate: Squaring Multimodal Coherence, Body Reference and Facial Biometrics. in: J. Pflaeging & J. Wildfeier & J. Bateman (eds.), Empirical Multimodality Research: Methods, Evaluations, Implications. Berlin/Boston: de Gruyter, 141-158.

Sloetjes, H. & Wittenburg, P. (2008): Annotation by Category: ELAN and ISO DCR. in: ELRA - European Language Resources Association (ed.), Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008). (no pages)