I will examine the common ground between artificial intelligence and multimodal analysis, drawing upon discussions at a workshop organised by the University of Cambridge and University College London. I will discuss the challenges faced by both fields of research and how they can both benefit from direct engagement with each other. I will also examine the challenges which arise when the two fields do intersect, drawing upon my experience of developing the Multimodal Analysis Platform (MAP), a cloud-based platform for collecting, storing and analysing news media and social media at scale (O’Halloran, Pal and Jin 2021). MAP was developed in order to integrate disparate modalities (text, image and videos) from multiple data source streams using the latest technologies in data science, big data, NLP and image processing. Case studies undertaken using MAP demonstrate (a) the benefits of large-scale analysis and (b) the current limitations of big data approaches to multimodal analytics. In particular, I will discuss how a context-based approach and knowledge about language, images and other resources as semiotic systems can contribute to transparent AI techniques.

O'Halloran, K. L., Pal, G., & Jin, M. (2021). Multimodal approach to analysing big social and news media data. Discourse, Context & Media, 40, doi.org/10.1016/j.dcm.2021.100467

BIO NOTE

Professor Kay O’Halloran is Chair Professor and Head of Department of Communication and Media in the School of the Arts at the University of Liverpool and Visiting Distinguished Professor at the Martin Centre for Appliable Linguistics at Shanghai Jiao Tong University, China. Prior to this, she worked at Curtin University, Western Australia (2013–2019), and the National University of Singapore (1998–2013) where she was a member of the Department of the English Language & Literature, and Director of the Multimodal Analysis Lab in the Interactive & Digital Media Institute. Kay is an internationally recognized academic in the field of multimodal analysis, involving the study of the interaction of language with other resources in texts, interactions and events. A key focus of her work is the development of digital tools and techniques for multimodal analysis, in particular the development of multimodal mixed methods approaches for big data analytics.