Pre-conference Panel: A comprehensive approach to developing an ASL test 

A comprehensive approach to developing an ASL test: DEI, validity, reliability, and practicality

Moderator: Meg Malone (ACTFL)

Panel participants: Camelot Marshall (ACTFL), Carly Favero (ACTFL), Hannah Lukow (Penn State), Shay Taylor-El (Gallaudet University)

• Location: Intercultural Center Auditorium

• Date: Thursday, October 24, 2024 at 4:00 p.m.

ASL interpretation will be available for the panel. To register, please select it as a ticket add-on while registering for ECOLT.

Abstract

ASL has been under-represented in language assessment; the diversity of ASL users (L1, children of Deaf adults (CODAs), and world language learners) makes developing and administering ASL tests a challenging and significant equity issue. ASL test design and delivery require input from a variety of experts to ensure a reliable and valid product. This work was steered by an external advisory committee of high school and university-based ASL educators, assessment and curriculum experts, interpreters, and a film production team.

This panel describes the iterative process to develop a computer-based ASL test for Receptive Comprehension (RC) and Interpersonal Interaction (II) and includes reflections from all contributors. Phase 1 of the project established an advisory committee to determine test specifications. Phase 2 focused on developing and adapting test tasks appropriate for the intended audience and filming initial test content. Phase 3 (world language learners of ASL) included a pilot study that examined the affordances of existing technology for test delivery. This phase also explored the delivery system’s ability to accurately employ multiple-choice responses to RC signed tasks and record learner’s responses to video prompts. Data collection included background surveys, observations, and stimulated recalls. The data were then reviewed by ASL subject matter experts (N=5) to determine the instrument’s usefulness and recommend improvements. Phase 4 included formal production of the test content and a second pilot study to evaluate the instrument’s effectiveness in assessing RC as well as eliciting language during the II section. During this phase, the participant pool (N=60) was expanded to include L1, CODAs, and world language learners. Phase 5 addresses rating requirements, and rubric creation to facilitate rating the participants’ recorded responses. The panel will show how experts collaborated to develop a test to meet the ASL community’s needs, while adhering to best practices in test development.