[Context] In a context of increased use of multimodal supports for information transfer, it is relevant to ask ourselves how one can (i) analyze, (ii) generalize results, and (iii) visualize results of multimodal analysis. One way of achieving this program could be to perform a constructional analysis. In this regard, a construction is to be defined as a function-form couple (Goldberg 2006). Constructions have been theorized since the 1990s with a monomodal and syntactic perspective. If we however abstract what the function and the form of an information can be, one could hypothesize that every single information exchanged during a human conversation is a construction (Tomasello 2003). This hypothesis will be followed in the proposed talk and will be the basis of a discussion around the theoretical and methodological requirements needed for making Construction Grammar multimodal.

[Data] The proposed discussion is based on examples collected in authentic communicative settings, which are structured in corpora. Two corpora will be used:

1. The first one is made of 908 advertisements published in supermarket flyers in Austria between 2017 and 2018 (fully representative of the ads for wine discourse, see Bach 2021 for details);

2. The second corpus is a compilation of Internet-Memes (also used in Bach 2021) derived from the “disloyal man” and others from the cycling community.

[Aim] The aim of the talk is not to propose a detailed analysis of a multimodal category, neither is it to compare both corpora. The purpose of the presented study is to think Construction Grammar from a multimodal perspective based on real-life examples.

[Discussion] After having introduced the talk and defined the theoretical background of Construction Grammar, it will be shown that a multimodal content is a construction. Then examples will be discussed and it will be underlined that the visual part of the information is irrelevant in the organization of the function of the information. Nevertheless, it will be shown that the issue does not come from the theory but from the information it-self. The advertisement is much more a textual structure with a picture as illustration rather than a fully multimodal structure. Precisely, other examples like Internet memes (Bülow/Merten/Johann 2018) are fitter for developing such an approach and can indeed improve the theory in its totality. Therefore, the answer to the question “Do we really need a multimodal construction grammar” (Ziem 2017) is definitely a yes. However, the main problem is how construction grammar should integrate multimodal components. This talk takes part of the ongoing discussion (see Zima/Bergs 2017) and tries to underline what should be done. The end of the talk is dedicated to a summary of the argumentation and to some inputs for technology transfer actions, especially in the exchange of information between human and robot (or AI-system).

Bach, Matthieu (2021): Vers une Sémantique Discursive cognitive. Réflexions théoriques et applications empiriques sur un corpus de langue allemande [PhD Thesis]. Dijon: UBFC.

Goldberg, Adele E. (2006) : Constructions at work. The nature of generalization in language. Oxford et al.: Oxford University Press.

Bülow, Lars / Merten, Marie-Luis / Johann, Michael (2018) : “Internet-Memes als Zugang zu multimodalen Konstruktionen.” In: Zeitschrift für Angewandte Linguistik 69, 1–32.

Tomasello, Michael (2003) : Constructing a Language: A Usage-Based Theory of Language Acquisition. Cambridge/London: Harvard University Press.

Ziem, Alexander (2017): “Do we really need a Multimodal Construction Grammar?” In: Linguistic Vanguard 3(1), 1–9.

Zima, Elisabeth/Bergs, Alexander (eds.) (2017): “Towards a multimodal construction grammar.” In: Linguistic Vanguard 3(1).