MULTIMODALITY IN WRITTEN ACADEMIC GENRES: AN INTEGRATED CORPUS- AND COMPUTER-AIDED TEXTUAL ANALYSIS OF IMAGE-TEXT RELATIONS IN ESP STUDENT WRITING
While several studies (Royce 2002, Guo 2004, Unsworth 2007) have highlighted the centrality of image-text relations in educational settings, research has yet to systematically investigate the co-deployment of textual and visual resources in written academic genres. This study therefore proposes a framework for analysing multimodal meaning making in L2 English writing in an ESP setting. Drawing on Systemic Functional Linguistics (Halliday, 1994; Martin 1992), the study of multimodality (Liu & O’Halloran 2009) will be integrated with a discourse-semantic approach (Martin & Rose 2007) to written academic genres. I will argue that, while ‘image-text relations’ (Martinec & Salway 2005, Stöckl 2011, Bateman 2014) are central to meaning making in ESP writing, they pose a particular challenge for L2 student writers. This research is driven by three questions: What image-text relations can be identified in the data? How do these resources contribute to textual meaning making? To what extent are business students familiar with multimodal practices? Building on a self-compiled specialised corpus, this study proposes an integrated approach combining corpus analysis and qualitative data analysis (QDA). A major advantage of relating text to corpus evidence is that it provides the researcher with two entry points into the data. In first adopting a text-based perspective, QDA tools are used to code the various image-text relations in the corpus data. Subsequently, corpus methodology is used to retrieve co-occurrence patterns of textual and visual resources. The findings show great variation in business students’ multimodal meaning-making abilities. A case in point is these writers’ use of visuals as either expository, illustrative or self-explanatory. Results also showed that these apprentice writers seem to be lacking in the necessary semiotic resources to produce meaningful writing. The importance of multimodality in academic and professional practice calls for a shift in ESP instruction from language-focused to multimodal pedagogy.
Bateman, John A. (2014): Text and Image: A Critical Introduction to the Visual/Verbal Divide. London: Routledge.
Guo, L. (2004): Multimodality in a biology textbook. in: K. L. O'Halloran (ed.), Multimodal Discourse Analysis: Systemic-Functional Perspectives. London: Continuum, 196-219.
Halliday, Michael A. K. (1994): An Introduction to Functional Grammar. Second edition. London: Arnold.
Liu, Y. & O'Halloran, K. L. (2009): Intersemiotic texture: Analysing cohesive devices between language and images. Social Semiotics 19 (4), 367-388.
Martin, J. R. (1992): English Text: System and Structure. Philadelphia: Benjamins.
Martin, J. R. & Rose, D. (2007): Working with Discourse: Meaning beyond the Clause. Second editiion, reprinted. London: Continuum.
Martinec, R. & Salway, A. (2005): A system for image-text relations in new (and old) media. Visual Communication 4 (3), 337-371.
Roye, T. (2002): Multimodality in the TESOL classroom: Exploring visual-verabl synergy. TESOL Quarterly 36 (2), 191-205.
Stöckl, Hartmut (2011): Sprache-Bild-Texte lesen. Zur Methodik einer Grundkompetenz. in: H.-J. Diekmannshenke, M. Klemm & H. Stöckl (eds.), Bildlinguistik: Theorien - Methoden - Fallbeispiele. Berlin: E. Schmidt, 43-70.
Unsworth, Len (2007): Image-text relations and intersemiosis Towards multimodal text description for multi-literacies education. in: L. Barbara & T. Berber Sardinha (eds.), Proceedings of the 33rd International Systemic Functional Linguistics Congress. Sao Paulo, Brazil: PUCSP, 1165-1205. https://www.pucsp.br/isfc/proceedings/Artigos%20pdf/59pl_unsworth_1165a1205.pdf