Assessment Creation Activity 2: Listening

CONTEXT OF ASSESSMENT

This course is the first term of first year Russian as a foreign language at an American university, such as University of Oregon. Students have all graduated from an American high school. It is presumed that their L1 is English and their L2 is Russian, but in reality students could have been exposed to other languages either as their L1 or L2. University of Oregon has a minimum two-year foreign language requirement for students graduating with a BA. Russian is not mandatory - students can choose from a variety of languages - so students could be both extrinsically and intrinsically motivated. First-year Russian classes at UO usually have 16-18 students. Students meet in the mornings, 5 days a week, for 50 minutes each class session. The course is 10 weeks long, so students receive 50 hours of instruction.

PURPOSE OF ASSESSMENT & CONSTRUCTS MEASURED

Some of the overall course objectives are: SWBAT ask and answer questions about topics of daily life at the novice level. SWBAT show understanding of 500 high frequency words. By this time in the course, students have become accustomed to listening to short texts and answering questions. The purpose of this assessment is to measure listening comprehension at the novice level.  This is a formative, criterion-referenced, not graded (but completion/participation points and written feedback will be given). It will assess listening comprehension of new vocabulary words learned in that unit. This unit focuses on talking about possessions and negation. Numbers were also covered in this term. Vocabulary about rooms, common objects in rooms, and colors was covered earlier in the term.

In this assessment, students will be encouraged to think about the following ACTFL Can-Do statement, which reflects novice interpretive proficiency, and how much it describes their proficiency: “I can identify the general topic and some basic information in both very familiar and everyday contexts by recognizing practiced or memorized words, phrases, and simple sentences in texts that are spoken, written, or signed” (www.actfl.org). This assessment will lead up to students practicing speaking to people on the phone while searching for an apartment.

Students will be tested on the following marcroskills:

• Selective listening for content/details

• Recognize new vocabulary learned that week

• Recognize negation

TASK USED TO MEASURE SKILLS & CONSTRUCT

The assessment will be done through information transfer (multiple-picture-cued selection). It is written by the instructor to represent a situation where someone describes their dream home. The instructor reads the text to the students. Students are presented with three pictures per question. They mark the one that is being described by the teacher. There are 5 questions total. The teacher will read the text twice, from start to finish, at a slow pace, since students are at a novice-low to novice-mid level.

TEXT PROMPT

Я иногда говорю о доме моей мечты в разговоре с близкими мне людьми. 

Дом моей мечты не большой, но красивый и уютный. Он белый и в нём два этажа.

Мне очень нравятся книги, поэтому я хочу, чтобы в моем доме была библиотека и камин для чтения в гостинной.

Я хочу, чтобы из кухни можно было выйти на задний двор, поэтому на кухне будут стеклянные двери. 

Мне не нравятся большие спальни - я хочу уютную спальню, может быть с желтым одеялом.

В ванной комнате я хочу большую белую ванную и большое окно. 

Но самое главное, я хочу, чтобы в моем доме были близкие мне люди - моя семья, с которыми можно и плакать, и радоваться.

(Translation:  I sometimes talk about my dream home in conversations with people close to me.

The house of my dreams is not big, but beautiful and cozy. It is white and has two floors.

I really like books, so I want to have a library in my house and a fireplace for reading in the living room.

I want the kitchen to have access to the backyard, so the kitchen will have glass doors.

I don't like big bedrooms - I want a cozy bedroom, maybe with a yellow duvet.

In the bathroom, I want a big white bathtub and a big window.

But most importantly, I want people close to me in my house - my family, with whom I can both cry and rejoice.)

Correct answers are highlighted for the purpose of this LT 549 assignment. 

INSTRUCTIONS  

Look at the images below while you listen to your teacher read about their dream home. 

For each question, circle the letter next to the image that matches the house your teacher is describing.

BENEFITS OF THESE TASKS

• Reliability - if the spoken part and the picture options are well thought through, test scores should be highly consistent.

• Construct validity - if the goal is to test listening only, this assessment does that because students do not need to read, write or speak.

• Content validity - the assessment covers what was learned in that unit, which are vocabulary words related to numbers and colors, grammar related to expressing possessions, and there is one sentence about negation. Vocabulary about rooms, common objects in rooms, and colors was covered earlier in the term.

• Practicality - this text was relatively easy to create and should be straightforward for students to complete.

• Authenticity - even though the text is created by the instructor, it is a real-world situation, as people are very likely to talk about their ideal home. People also look at places like Pinterest for inspiration for their dream home, so looking at several pictures is realistic.

DRAWBACKS OF THESE TASKS

• Reliability - as with all multiple choice questions, it is very difficult to create successful items. Before piloting this test, it is difficult for me to tell how well the distractors are working in my assessment. Guessing can also account for some correct answers, which can lead to negative washback. Finally, how slow or fast I read the text can have an effect on student responses.

• Construct validity - if the goal is to test integrative skills, this assessment would not work.

• Practicality - even though the assessment was relatively “teacher-friendly”, practicality decreases because of the difficulty of creating successful items and distractors.

To increase reliability, I could record the listening prompt, so that all students across classes hear it at the same speed. I am also planning on having students discuss their experience (what was easy to understand, what was confusing, etc.) with their peers after I grade the assessment. Discussions could also be done as a class, so that I could benefit from their feedback and modify as needed. The prompt might need to be modified, or the pictures, depending on what students say. Students could also get a teaching moment through discussions with peers.

OUTLINE OF HOW OTHERS CAN CREATE A SIMILAR ASSESSMENT

I went through many authentic Youtube videos where people give a house tour in Russian, but they speak too fast, quiet, or unclear for novice learners to understand. Thus, an instructor-dictated text is more appropriate at this stage. The text has to be relatively short in order to avoid test-taker fatigue. To increase authenticity, it would be important for the instructor to give enough background information about the listening prompt. It can be an authentic text, but possibly modified to the level of novice learners, and it must include topics and vocabulary already covered. 

I also checked several podcast and video excerpts for vocabulary size statistics, and found that it is difficult to find a text that has a high percentage of top 2000 words. All of the texts I checked had coverage less than what is required for readability (which is 95%-98%), even though they were rated from “very easy” to “moderately easy” on  http://ru.readability.io/ . The following are statistics for the prompt used in this assessment:

Assessment Creation Activity 5: Pragmatics

CONTEXT OF ASSESSMENT

This course is the first term of first year Russian as a foreign language at an American university, such as University of Oregon. Students are graduates of American high schools. It is presumed that their L1 is English and their L2 is Russian, but in reality students could have been exposed to other languages either as their L1 or L2. University of Oregon has a minimum two-year foreign language requirement for students graduating with a BA. Russian is not mandatory - students can choose from a variety of languages - so students could be both extrinsically and intrinsically motivated. First-year Russian classes at UO usually have 16-18 students. Students meet in the mornings, 5 days a week, for 50 minutes each class session. The course is 10 weeks long, so students receive 50 hours of instruction.

PURPOSE OF ASSESSMENT & CONSTRUCTS MEASURED

This assessment is used to measure pragmatic proficiency at the novice level. It is an achievement assessment that is formative. It is criterion-referenced and informal, as it will not be graded. Students will have the opportunity to discuss the decisions they made on the assessment with the class, in a group, with a partner, and/or with the instructor. The following are overall course goals that are addressed in this assessment: 

• SWBAT express greetings, apologies, likes/dislikes and preferences in a culturally appropriate way.

• SWBAT ask and answer questions about topics of daily life at the novice level.

• SWBAT compare and contrast some of their cultural practices to those of the target culture.

The following skills will be tested:

• Recognition and production of routine formulae (formal vs. informal address)

• Evaluation of dialogic interactions (contextualization cues, effect on interlocutor, repair)

I believe it is important for students to feel comfortable with the pragmatics of using formal and informal options of "you". In Russia and other places where Russian is spoken, usually, "Вы" is used with strangers, bosses, teachers, and people older than the speaker, in general. "Ты" is general used with family, friends, and people younger than the speaker. However, there are times when the formal "you" is used with family, like aunts and uncles. There are also instances when one would start with the formal address and quickly change to informal.

In this assessment, I would like to focus on navigating "you" with strangers, when speakers start with the formal and then have the option of continuing using the formal, making a request to transition to the informal and going through with it, or switching to the informal without a request. All of these would be appropriate, depending on the person and the situation. The inappropriate option would be to start with the informal address. However, even in this case, one could apologize and continue with the formal, if that is what the other person wants.

TASKS USED TO MEASURE SKILLS & CONSTRUCTS

This task will be mostly in English, and not the target language because students have not yet learned the complex language that is required to explain this concept that they must know as novice learners. Students will be provided with a test page that has four scenarios of adults speaking to each other. The scenarios will be in English because students will most likely not be at the proficiency level to be able to read them in the target language. Students choose the appropriate decision for each scenario. Students will work independently on this assessment, but afterwards, maybe in the next class session, the class together (or possibly in small groups) would justify their decisions.

OUTLINE FOR HOW OTHERS CAN CREATE A SIMILAR ASSESSMENT

To recreate this task, one would need to think of situations in which someone talks to a stranger or an acquaintance. It would be very important to create scenarios that are realistic and relevant to each group of learners. This activity can also be made to assess the pragmatics of formal and informal address with anyone, not just strangers. The instructions would have to be clear so that students know how they are expected to answer. I chose to provide options for the answers, but they can be left out for more advanced speakers.