What is a Multimodal Text?

Multimodal Texts: A Semiotics-Based Definition

A multimodal text can refer to interactive objects or interrelational systems depending on the discipline that is defining the term. Anstey and Bull (2010) offers a pedagogically useful and semiotics-based definition of a multimodal text:

"A text may be defined as multimodal when it combines two or more semiotic systems. There are five semiotic systems in total:

1. Linguistic: comprising aspects such as vocabulary, generic structure and the grammar of oral and written language

2. Visual: comprising aspects such as colour, vectors and viewpoint in still and moving images

3. Audio: comprising aspects such as volume, pitch and rhythm of music and sound effects

4. Gestural: comprising aspects such as movement, speed and stillness in facial expression and body language

5. Spatial: comprising aspects such as proximity, direction, position of layout and organisation of objects in space.

Examples of multimodal texts are:

• a picture book, in which the textual and visual elements are arranged on individual pages that contribute to an overall set of bound pages

• a webpage, in which elements such as sound effects, oral language, written language, music and still or moving images are combined

• a live ballet performance, in which gesture, music, and space are the main elements.

Multimodal texts can be delivered via different media or technologies. They may be live, paper, or digital electronic."

-- Michèle Anstey and Geoff Bull

How Do We Analyze the Semiotic Elements of a Multimodal Text?

In traditional literary analysis, interpreting a linguistic-based text, such as a poem, a short story, a novel, and other types of written texts, means to dissect the written text by its words and symbols in order to uncover its overall meaning. We are asked to only "wrestle" with the text's linguistic semiotic element. On the other hand, multimodal texts are a bit more complicated to understand because they contain 2 or more of the 5 semiotics elements (linguistic, audio, visual, gestural, spatial), meaning that we would have to wrestle with multiple semiotic elements to uncover the textual meaning. The question we need to address is how does each semiotics element influence the meaning we infer from a text? How do the semiotics elements work with each other to influence our understanding of a text?

To understand how to analyze a multimodal text, let's examine the lyrics (linguistic element) to the song, "Love Is Bigger Than Anything In Its Way" by U2.

Lyrics (Linguistic Element)

"Love Is Bigger Than Anything In Its Way" by U2

The door is open to go through

If I could I would come, too

But the path is made by you

As you're walking start singing and stop talking

Oh, if I could hear myself when I say

(Oh love) love is bigger than anything in its way

So young to be the words of your own song

I know the rage in you is strong

Write a world where we can belong

To each other and sing it like no other

Oh, if I could hear myself when I say

(Oh love) love is bigger than anything in its way

If the moonlight caught you crying on Killiney Bay

Oh, sing your song

Let your song be sung

If you listen you can hear the silence say

"When you think you're done

You've just begun"

Love is bigger than anything in its way

Love is bigger than anything in its way

Love is bigger than anything in its way

Songwriters: Adam Clayton / Dave Evans / Larry Mullen / Paul David Hewson

-------------------------------------------------------------------

Q. What are the possible meanings of the "Love Is Bigger Than Anything In Its Way?" lyrics? In your answer, identify specific words in the text that support your interpretation of the lyrics.

Now, let's take a listen to the "Love Is Bigger Than Anything In Its Way" music video (audio + visual elements) accompanied by the lyrics (linguistic element).

Music Video (Audio + Visual Elements)

Q. What is the difference in meaning between the meaning of the lyrics (linguistic) and the meaning of the lyrics (linguistic) with the additional song (audio) and music video (visual) of U2's "Love is Bigger Than Anything In Its Way"?

-------------------------------------------------------------------

Let's examine another set of lyrics for the song, "Evil" by Interpol.

Lyrics (Linguistic Element)

"Evil" by Interpol

Rosemary

Heaven restores you in life

You're coming with me

Through the aging, the fearing, the strife

It's the smiling on the package

It's the faces in the sand

It's the thought that holds you upwards

Embracing me with two hands

Right will take you places

Yeah maybe to the beach

When your friends they do come crying

Tell them now your pleasure's set up on slow-release

Hey wait

Great smile

Sensitive to faith, not denial

But hey whose on trial?

It took a life span with no cellmate

The long way back

Sandy, why can't we look the other way?

We speaks about travel

Yeah, we think about the land

We smart like all peoples

Feeling real tan

I can take you places

Do you need a new man?

Wipe the pollen from the faces

Make revision to a dream while you wait in the van

Hey wait

Great smile

Sensitive to faith, not denial

But hey whose on trial?

It took a life span with no cellmate

To find the long way back

Sandy, why can't we look the other way?

You're weightless, you are exotic

You need something for which to care

Sandy, why can't we look the other way?

Leave some shards under the belly

Lay some grease inside my hand

It's a sentimental jury

And the makings of a good plan

You've come to love me lightly

Yeah you've come to hold me tight

Is this motion ever lasting

Or do shudders pass in the night?

Rosemary

Oh heaven restores you in life

I spent a lifespan with no cellmate

The long way back

Sandy, why can't we look the other way?

You're weightless, semi-erotic

You need someone to take you there

Sandy, why can't we look the other way?

Why can't we just play the other game?

Why can't we just look the other way?

Songwriters: Charles Edward Daniels

-------------------------------------------------------------------

Q. What are the possible meanings of the "Evil" lyrics? In your answer, identify specific words in the text that support your interpretation of the lyrics.

Now, let's take a listen to the "Evil" music video (audio + visual elements) accompanied by the lyrics (linguistic element).

Music Video (Audio + Visual Elements)

Interactive Images

Let's examine an interactive image, which is a type of multimodal text, to understand how our understanding of the world is influenced by multimodality.

"Monsters on the Midnight Train"

Q. What are the semiotics elements present in the above interactive image? How do the semiotics elements influence our understanding of what is occurring in the image? How do the image speak to certain cultural and social conventions and codes? How do the cultural and social conventions support or interfere with our understanding of the image?

Poems (Written, Spoken, Visualized)

Poems can also be constructed in different multimodalities. We need to examine a collection of poems, composed in different multimodalities, to understand how the poems shift meaning based on their construction. First, let's read a poem written by poet, Dylan Thomas.

Written Poem: "Do not go gentle into that good night" by Dylan Thomas, 1914-1953

Do not go gentle into that good night,

Old age should burn and rave at close of day;

Rage, rage against the dying of the light.

Though wise men at their end know dark is right,

Because their words had forked no lightning they

Do not go gentle into that good night.

Good men, the last wave by, crying how bright

Their frail deeds might have danced in a green bay,

Rage, rage against the dying of the light.

Wild men who caught and sang the sun in flight,

And learn, too late, they grieved it on its way,

Do not go gentle into that good night.

Grave men, near death, who see with blinding sight

Blind eyes could blaze like meteors and be gay,

Rage, rage against the dying of the light.

And you, my father, there on the sad height,

Curse, bless, me now with your fierce tears, I pray.

Do not go gentle into that good night.

Rage, rage against the dying of the light.

Audio Poem: "Do not go gentle into that good night" by Dylan Thomas, 1914-1953

Written Poem: The Road Not Taken by Robert Frost, 1874-1963

Two roads diverged in a yellow wood,

And sorry I could not travel both

And be one traveler, long I stood

And looked down one as far as I could

To where it bent in the undergrowth;

Then took the other, as just as fair,

And having perhaps the better claim,

Because it was grassy and wanted wear;

Though as for that the passing there

Had worn them really about the same,

And both that morning equally lay

In leaves no step had trodden black.

Oh, I kept the first for another day!

Yet knowing how way leads on to way,

I doubted if I should ever come back.

I shall be telling this with a sigh

Somewhere ages and ages hence:

Two roads diverged in a wood, and I—

I took the one less traveled by,

And that has made all the difference.

Audio Poem: The Road Not Taken by Robert Frost, 1874-1963

Visual Poem: The Road Not Taken by Robert Frost, 1874-1963 (Illustrated by Zen Pencil)

Visual Art (Interactive Art Objects)

Cabbibo

"Universe of Sound" by Cabbibo

"Burial" by Cabbibo

"Pattern" by Cabbibo

"Sol" by Cabbibo

Visual Art (Music Videos)

What K-Pop Can Teach Us About Design

Music Videos from the East

Music Videos from the West