Are large multimodal corpora with rich annotations a pipe dream?

Diverse research communities have considered large annotated corpora highly beneficial - or even necessary - for achieving a more comprehensive understanding of multimodality as a phenomenon (Bateman 2014). This may be traced back to the field of linguistics and how the introduction of large corpora and the subsequent development of powerful methods for their analysis allowed conducting empirical research on language at an unprecedented scale.

In this presentation, I seek to map the state-of-the-art in corpus-driven research on multimodality, focusing particularly on issues related to the scale of multimodal corpora and the depth of annotations, that is, to what extent the features of the communicative situations or artefacts are described by the corpus.

More specifically, I seek to address three questions that I consider crucial for advancing corpus-driven research on multimodality: 

(1) How to achieve sufficient depth while increasing scale?

(2) How does the lack of established methods for multimodal corpus analysis affect the use of corpora?

(3) Are linguistic corpora ultimately a suitable yardstick for multimodal corpora?

Finally, I discuss how emphasising scale or depth might affect our ability to pursue analyses of multimodality. I argue that we need to find a balance between these two issues, while simultaneously considering to what extent the annotation may be automated or supported using computational methods, while also keeping the limitations of such methods in mind (Hiippala 2023, 2024).

References

Bateman, John A. 2014. Using multimodal corpora for empirical research. In Jewitt, Carey (ed.) The Routledge Handbook of Multimodal Analysis. Second edition. Routledge: London and New York, pp. 238–252. 

Hiippala, Tuomo. 2023. Corpus-based insights into multimodality and genre in primary school science diagrams. Visual Communication. doi: 10.1177/14703572231161829

Hiippala, Tuomo. 2024. Rethinking multimodal corpora from the perspective of Peircean

semiotics. Frontiers in Communication 9. doi: 10.3389/fcomm.2024.1337434