Supervision

Students will be guided by the technical team throughout the whole process. Main steps towards the completion of the BA thesis (TFG):

1. • Identify a hypothesis on gesture or speech variation for semantic discrimination. E.g., some features of gestural behavior, or in the intonation or intensity of speech, might show significant and systematic differences when co-occurring with yesterday/today/tomorrow, or before/after. Time expressions will be preferred, but it is also possible to examine gesture co-occurring with other semantic fields or linguistic functions, such as negation, agreement and disagreement, discourse transitions, and more. The theoretical relevance of the hypothesis and the innovativeness of the methods will be assessed through an appropriate discussion of the state of the art.

   • Build and annotate a video corpus with multiple clips of people uttering the words or phrases selected for the study. You will search and annotate data from the NewsScape Library of TV News. The student will use the Erasmus Plus MULTIDATA platform, plus additional tools developed by the Daedalus Lab and the Red Hen Lab. Annotation will be mostly carried out automatically. Some manual annotation may be required, but it will not be time-consuming. The result will be a corpus including hundreds of 5-10 seconds video clips in which a speaker can be seen uttering the target linguistic expression.

   • Description of the dataset and initial preliminary statistics related to the variables in the hypothesis. Write up a detailed description of the methods that led to the construction of the corpus. Carry out analyses involving descriptive statistics and report the results. Write a discussion.

   • Elaborate a model to explain variation patterns in the data. This part is optional for a BA thesis. Make an informed decision on the choice of statistical techniques for analyzing the data. Aided by ChatGPT or a similar resource, carry out the modeling and report and discuss the results.

This topic involves collaborations within the MULTIFLOW project: Raúl Sánchez, Senior Computer Scientist at the Faculty of Computer Science; Brian Herreño, Data Science Researcher at the MULTIFLOW project; Peter Uhrig (Professor) and Armine Garibyan (Postdoctoral Researcher) from the Chair of Digital Linguistics with a focus on Big Data at Friedrich-Alexander-Universität Erlangen-Nürnberg. It is also possible to collaborate with Inés Olza, Director of the Multimodal Pragmatics Lab, Institute for Culture and Society, University of Navarra.

Use off-the-shelf open software solutions to address a variety of challenges in multimodal communication analysis. This topic may require a certain degree of technical expertise, typically involving computer science students, from the Faculty of Computer Science at the University of Murcia or recruited through Google Summer of Code. Nevertheless, Humanities students with little or no technical background also work on testing and evaluating these tools.

At our Google Summer of Code page you can find some of our current ideas. Some areas of interest:

• Automatic speech recognition: e.g. WhisperX.

• Acoustic analysis of speech, phonetics, prosody: e.g. PRAAT

• Computer vision: key bodypoint detection (e.g. OpenPose), automatic video description, automatic object recognition, automatic speaker detection

• Reconstructing gesture trajectories in 2D and 3D

• Testing statistical and machine-learning techniques on multimodal datasets

• Integrating and streamlining various tools into a multimodal processing pipeline

• Developing specific tools and processes for the digitization and analysis of intangible cultural heritage (oral traditions, music, linguistic fieldwork, etc.)

This topic involves collaborating with Raúl Sánchez, Senior Computer Scientist at the Faculty of Computer Science; Brian Herreño, Data Science Researcher at the MULTIFLOW project; Rosa Illán, Postdoctoral researcher at the University of Lyon and CNRS; Peter Uhrig (Professor) and Armine Garibyan (Postdoctoral Researcher) from the Chair of Digital Linguistics with a focus on Big Data at Friedrich-Alexander-Universität Erlangen-Nürnberg. It may also involve the Red Hen Lab and the Language Archive and the e-Humanities Department at the University of Cologne.

Build on the activities and resources of the Erasmus Plus MULTIDATA platform to develop a teaching experience or didactic materials connecting multimodality to topics in:

• language learning

• linguistics

• media studies

• literary and film studies

• psychology

• cognitive science

• STEM

MULTIDATA facilitates the obtaining of multimodal data (gesture, posture, facial expression, speech transcription and audio analysis, and more) from videos in multiple languages. You will propose a teaching project, test its effectiveness, and reflect on its design. You can also study the current state of multimodality across subjects in higher education and propose actions based on the results of your research and on the possibilities offered by MULTIDATA.

This topic involves collaboration with the MULTIDATA team: Daniel Alcaraz, Beatriz Galindo Professor, Daedalus Lab, University of Murcia; Irene Bolumar, PhD candidate and Fulbright-Séneca Fellow, Daedalus Lab, University of Murcia; Raúl Sánchez, Senior Computer Scientist at the Faculty of Computer Science. Interactions with the MULTIDATA teams at FAU Erlangen-Nürnberg, Radboud Nijmegen, and the Max Planck Institute for Psycholinguistics are also possible. Collaboration with the Red Hen Lab is probable.

Study communication-related challenges and opportunities experienced by cooperation organizations working with people with severe disabilities in unfavorable environments. Topics may include:

1. • Challenges in intercultural and multimodal communication for health and social care connected to disability: NGO volunteers, indigenous communities, war veterans, paralympic athletes.

   • Multimodal communication and language technologies: their impact on disability.

   • Multimodal and media communication for NGOs working with people with disabilities: social media and other communication strategies, oral communication, rhetoric, advertising, and more.

   • Teaching English, multimodal communication, and communication for healthcare professionals as part of the cooperation actions of NGOs focusing on disabilities.

This topic involves collaboration with the Hands with Heart Foundation. It is also possible to collaborate with the Diversity Services at UMU. The work can be done without traveling and we do not have funding for fieldwork associated with this topic. However, if you really want to be in the field and are passionate enough, you should not let the lack of funding, or anything else, stop you.

Systematic case studies of a particular multimodal technique or resource and its aesthetic and communicative effects. Video and audio data from different sources can be analyzed: television and film are preferred, but social media, radio, and other sources are also possible. Examples:

• Cognitive operations underlying creative processes across literature, music composition, and graphical or pictorial representations

• Transitions across film, television, literature, and everyday language

• Reference across film, television, literature, and everyday language

• Zoom effects across camera techniques and linguistic patterns

• Conceptual compression across literature, film, television, and the arts

• Multimodal poetics in the verbal art of oral traditions (see topic 8. Cognition and Poetics)

This topic may involve collaboration with Anna Bonifazi, Professor, and Pinelopi Ioannidou, PhD candidate, at the Department of Discourse Studies, University of Cologne. Also possibly with the Red Hen Lab.

Study a topic in contemporary politics, society, or culture through big-data from the media. The primary source will be data from the NewsScape Library of TV News, but other sources, such as social media, are also possible. These projects combine NLP and multimodal analysis tools to extract big data about a particular topic from video collections or similar sources.

This topic usually involves collaboration with the Red Hen Lab, maybe also with Inés Olza, Director of the Multimodal Pragmatics Lab, Institute for Culture and Society, University of Navarra.

This interdisciplinary topic connects with most of the others, using many of their methods and approaches, albeit focusing on the way individuals and cultures create, think about, and represent time. It also builds on the results of several projects on this topic over the years, funded by grants from the Marie Curie Actions, the Ministry of Universities, the Séneca Foundation of the Murcia Region, or the European Network of Institutes for Advanced Studies, among others. See CREATIME.

Main areas of interest:

• Time in the mind. Using theory of conceptual integration, image schemas, 4E and distributed cognition, psycholinguistic evidence, and other approaches, study various manifestations of temporal meaning and propose patterns and principles explaining their connections and differences. Bibliographical reviews and meta-analyses are also possible.

• Time and poetics. Study the literary expression of time, compare it with everyday language and with cross-cultural findings about time concepts, and find out what all this tells us about the integration of concepts, the construction of new ideas, creativity, and the imagination. These are the topics in which we have research ongoing:

o   Corpus studies of the figurative language of time. The corpus that we are building for lyric poetry in English, Spanish, and Greek (PoetiCog) can be used as an initial source to obtain data. It is possible to analyze the evolution of time concepts across different historical periods, literary styles or genres, works of an individual author, and so forth. It is also possible to contribute to the construction of PoetiCog itself.

o   Time in a particular literary work or author. Students can benefit greatly from the ongoing research on Four Quartets, by T. S. Eliot, and Momo, by Michael Ende, but they can also propose other works.

• Time and multimodality. Comparative study of the expression of time across different modalities in communication. Ongoing research:

o   Time expression in speech and gesture. Typically, this type of study will take the shape described for topic 1 (analyzing variation patterns across gesture, speech, and language), with an emphasis on the relevance for research into time concepts. Bibliographical reviews and meta-analyses are also possible.

o   Time in music composition. Analyze the cognitive processes underlying the musical representation of time and compare them to other creative and everyday manifestations of the human mind, analyzing a variety of data: linguistic (composer’s writings, musicological studies and analyses, and more), graphical (sketches, graphs), technological (spectrograms), multimodal (video interviews with composers).

• Time in cinema, television, and the arts. Comparative study of a phenomenon related to the expression of time across the visual arts: e.g. flashbacks in cinema, temporal references in television, time compressions in painting.

This topic may involve collaboration with any of the members of the Daedalus Lab, with José Luis Besada (Department of Musicology at Universidad Complutense de Madrid and Radio Clásica, Radio Nacional de España), Inés Olza and Adriana Gordejuela (Universidad de Navarra), and other researchers in Spain or abroad.

Study the species-defining cognitive abilities that make humans what they are, through the analysis of their manifestations in the most creative uses of language. Investigate how the patterns of the imagination and the aesthetic effects of literary texts arise from those capacities of the human mind. You can use a corpus approach, exploiting and developing PoetiCog, the Red Hen Lab’s corpus of lyric poetry, or a method based on extensive or close reading. See POEMIND and FORMULEARN.

There is ongoing research on these areas:

• The poetics of interaction. Study how communication, interaction between persons or personalized entities, agency-causation relations, and other relevant aspects are expressed in literature. Analyze these phenomena in the light of theories and findings from cognitive science. What does this tell us about the nature of literature and of communication itself? Work usually involves the systematic analysis of various examples of fictive communication, indirect speech, dialog and conversation, and more.

• The Spatial Lyric: poetic imagery and the patterns of perception and attention. Study and classify expressions of emotions and other complex meanings in figurative language, mainly in lyric poetry, using theoretical background from conceptual integration, image schemas, and cognitive development. In particular, we analyze how basic spatial stories, which seem to be key for getting conceptual work started during infancy, are reused creatively for poetic purposes: containment, motion from A to B, obstacles, occlusion, and others.

• The cognitive science of oral-traditional poetry. This topic studies the formulaic and improvising nature of poetic creativity in oral-traditional performance, across traditions (yes, Homer and Beowulf, but also any other epic, lyric, or other genres, from around the world). We focus on:

o comparative phraseology and formulaic creativity, connecting with construction grammar and other usage-based theories from linguistics.

o   themes, stories, episodes, and narrative and lyric patterning, connecting with frame semantics, scripts from computer and information science, and other theoretical constructs from cognitive science

o   multimodal composition-in-performance, connecting with multimodality (study of gesture, gaze, prosody, melody, rhythm, and more) and with relevant theoretical approaches from cognitive science and philosophy (4E and distributed cognition, conceptual blending)

• Time and poetics (see topic 7. The cognitive science of time)

• Rethinking Figurativeness. Systematic studies, corpus-based or through extensive or close reading, of poetic imagery and motifs across traditions, using the theoretical background of conceptual blending and contrasting with other approaches, such as conceptual metaphor theory or relevance theory.