In many areas of multimodality research, there is an increasing awareness that it is beneficial to move beyond discussions of selected, or even individual cases. This makes natural contact with corpus-based approaches as they have developed in linguistics; and there are many techniques, principles and even tools that can be adapted for multimodal use. What then is raised as a challenge is just what properties, features or phenomena within potentially highly complex multimodal artefacts or performances should find their way into 'multimodal corpora' and how. In this presentation, I draw on our ongoing work on approaching multimodality across a broad range of media empirically by tightly relating description to theoretical categories, particularly categories emerging from our tristratal definition of semiotic modes in terms of material, form and discourse semantics. I suggest that making this connection explicit allows us to move beyond isolated descriptions of particular aspects of multimodal artefacts and performances both theoretically and practically in terms of coordinating larger-scale distributed research efforts. Given the huge diversity of semiotic resources that can be studied, such cooperative efforts are now essential for progress to be made beyond toy analyses. Particular examples I draw upon will include comics and graphic novels, educational videos, and diagrams.

BIO NOTE

John Bateman is Professor of English Applied Linguistics at Bremen University, Germany. He has been investigating the relation between language and other semiotic systems for many years, focusing on accounts of register, genre, lexicogrammatical description and theory, multilingual and multimodal linguistic description, and computational instantiations of linguistic theory. Current work involves multimodal semiotics and theories of discourse as well as the development of empirical methodologies for robust multimodal analysis. Publications include works on multimodality and genre (2008, Palgrave), text and image (2014, Routledge), and a textbook on multimodality (2017 in English and 2020 in German, de Gruyter, with Janina Wildfeuer and Tuomo Hiippala).