GENERATIVE ARTIFICIAL INTELLIGENCE TOOLS
Co-authored with Charlene Polio
This chapter discusses how generative artificial intelligence (GenAI) tools, particularly large language models (LLMs) like ChatGPT, are emerging as powerful web-based tools for research purposes, such as data analysis, in applied linguistics research. While much attention has focused on pedagogical applications, we review how GenAI can be leveraged to support various stages of the research processes in empirical studies, such as instrument design, automated coding, text annotation, and qualitative data analysis. We address key concerns around validity and reliability as well as ethical considerations related to transparency, data privacy, and potential bias in AI-generated output. Given that GenAI is in the early stage of research application, we describe its current capacities and limitations based on emerging empirical research and propose promising directions for future studies.
Co-authored with J. Elliott Casal
This study leverages concept-based language instruction (C-BLI) as an innovative pedagogical approach to teaching academic citation practices and broader concepts of intertextuality in academic research writing. Participants were 34 undergraduate students, predominantly multilingual English writers, enrolled in an advanced writing course at a Hispanic-serving US institution. The 4-week C-BLI workshop incorporated visualized conceptual aids to teach author prominence (i.e., integral vs. nonintegral citations) and authorial stance (i.e., reporting verbs). Moving beyond conventional rule-based instruction, the workshop emphasized applications of concepts in instructor-led and collaborative settings, verbalization of evolving understandings, and one-on-one interactions with the instructor. Data include students’ research papers before and after the intervention (with student revision rationales), instructor-led text protocol conferences, in-class activity data (e.g., concept identification, text analysis, and concept visualization), and written reflections. Students reported increased confidence and intentionality with target concepts, and changes in writing include a sharp increase in reporting-verb types, more dynamic use of integral and nonintegral citation structures, and lower reliance on direct quotations. Case studies highlight evolving conceptual understandings of target concepts, growing awareness of intertextuality as a social practice, and increased ability to articulate writing decisions. This study highlights the potential of C-BLI in fostering students’ engagement with citation as a rhetorical and agentive practice, addressing persistent challenges faced by student writers.
Co-authored with Paul Gibbons
Despite increased advocacy for translingual pedagogy, little research has explored how college students in mainstream academic writing classrooms—many of whom are multilingual, first-generation students—perceive this approach. Drawing on artifacts collected throughout one semester including surveys, metacognitive reflections, article annotations, and research papers, this study examines the perception of 38 students at a Hispanic-serving institution on the role of translingual pedagogy in their academic writing experiences. We found that while students reported a strengthened sense of identity and cultural expression, they often expressed significant discomfort and engaged in self-censorship when incorporating their home language(s) into academic writing. These challenges were particularly evident among first-generation, multilingual students who had internalized monolingual norms from an early age, as they had long been discouraged from using their home language(s) in academic settings. For these students, leveraging their full linguistic repertoire was both liberating and fraught with anxiety, as they feared deviating from standard English might compromise their academic legitimacy and disqualify them from participating in the professional world. We underscore the need for a nuanced translingual pedagogical framework that not only embraces linguistic diversity but also addresses the complex barriers students face when bringing their home language(s) and other language varieties into academic spaces.
LANGUAGE PLAY AS RESISTANCE: NAVIGATING THROUGH DIGITAL CENSORSHIP DURING THE COVID-19 PANDEMIC
Co-authored with Jiajun Liang
During the COVID-19 pandemic, increased measures of information control were implemented in China to manage the flow of sensitive information that may contradict the official narrative of national sacrifice, perseverance, and triumph. Building on previous scholarship on the subversive potential of translanguaging practices, the present study analyzes key moments during the initial outbreak of the pandemic, when Chinese netizens creatively combined linguistic and semiotic symbols to circumvent censorship measures. It demonstrates how these practices highlight the innovative and multimodal nature of netizens’ linguistic tactics and illustrate how translanguaging and trans-semiotizing both enabled and profoundly disrupted the process of meaning-making. This article shows that the use of creative code-meshing strategies is a powerful means of challenging the centralized governance of social media spaces in pursuit of freedom of expression.
OPTIMIZING AI FOR ASSESSING L2 WRITING ACCURACY: AN EXPLORATION OF TEMPERATURES AND PROMPTS
Co-authored with Charlene Polio and Adam Pfau
This study investigates the impact of temperature and prompt settings on ChatGPT-4 in assessing second language (L2) writing accuracy. Building on Pfau et al. (2023), we used a corpus of 100 essays by L2 writers of English and examined how three temperature settings (0, 0.7, 1) and two prompt types (defined, undefined) influenced ChatGPT-4’s performance in error detection compared to human coding. Results indicated that ChatGPT-4, while generally underestimating error counts compared to human coders, showed a strong positive correlation with human coding across various settings. Notably, prompts with a detailed definition of errors yielded higher correlation coefficients (ρ = 0.826 to 0.859) than those without (ρ = 0.692 to 0.702), suggesting that more detailed prompts enhance ChatGPT-4’s performance. Descriptive statistics showed that with a less-detailed prompt, the error detection ability of ChatGPT-4 was nearly identical across temperature settings, yet with a more detailed prompt, ChatGPT-4’s performance was slightly better at higher temperatures. We discuss the importance of temperature in relation to prompt specificity for reliable L2 writing accuracy assessment and provide suggestions for optimizing AI tools such as ChatGPT-4 for assessing L2 writing accuracy.
Co-authored with Jingyuan Zhuang, Ryan Blair, Amy I. Kim, Fei Li, Rachel Thorson Hernández, Luke Plonsky
Studies in Second Language Learning and Teaching, 2023
The importance of academic journals in second language (L2) research is evident on at least two levels. Journals are, first of all, central to the process of disseminating scientific findings. Journals are also critical on a professional level as most L2 researchers must publish articles to advance their careers. However, not all journals are perceived as equal; some may be considered more prestigious or of higher quality and may, therefore, achieve a greater impact on the field. It is therefore necessary that we understand the identity and quality of L2 research journals, yet very little research (e.g., Egbert, 2007; VanPatten & Williams, 2002) has considered these issues to date. The current study sought to explore L2 journal identity and quality, and the relationship between these constructs. In order to do so, a database was compiled based on three different types of sources: (1) a questionnaire eliciting L2 researchers’ perceptions of the quality and prestige of 27 journals that publish L2 research (N = 327); (2) manual coding of different types of articles (e.g., empirical studies, review papers), data (quantitative, qualitative, mixed), research settings, and authorship patterns (K = 2,024) using the same 27 journals; and (3) bibliometric and submission data such as impact factors, citation counts, and acceptance rates. Descriptive statistics were applied to explore overall quality and prestige ratings as well as publication trends found in each journal. The relationships between those patterns and subjective ratings were also examined. In addition, regression models were built to determine the extent to which perceptions of journal quality and prestige could be explained as a function of journal and article features. We discuss the findings of the study in terms of on-going debates concerning publication practices, study quality, impact factors, journal selection, and the “journal culture” in applied linguistics.
Co-authored with J. Elliott Casal
Journal of Second Language Writing, 2023
Since the 1960s, legal writing pedagogy in the United States has been heavily influenced by the plain English movement, which largely discourages the use of legal jargon and overly complex syntactic structures in legal documents. While these efforts to democratize legal English are laudable, it is unclear whether plain English practices can accommodate the needs of multilingual, foreign legal writers who have not yet acculturated to the U.S. legal discourse. Using mixed-effects modeling, this study tracks the development of syntactic and lexical complexity in 246 hypothetical legal essays written by 31 international Master of Laws students enrolled in a year-long legal language course. Results show that after one year of instruction, students demonstrated a significant increase in the use of nominalizations and sophisticated vocabulary. However, their sentences became notably shorter, and there was a marked decrease in the use of noun phrases with pre-modifying adjective within semesters. We discuss the intersection between plain English and academic writing practices and their impact on second language (L2) writers’ developmental trajectories. We also highlight important caveats to consider when implementing plain English pedagogy in legal language classrooms.
EXPLORING THE POTENTIAL OF CHATGPT IN ASSESSING L2 WRITING ACCURACY FOR RESEARCH PURPOSES
Co-authored with Adam Pfau and Charlene Polio
Research Methods in Applied Linguistics, 2023
This study investigates ChatGPT's potential for measuring linguistic accuracy in second language writing for research purposes. We processed 100 L2 essays across five proficiency levels with ChatGPT-4 and manually coded for precision and recall with regard to ChatGPT's identification of errors. Our findings indicate a strong correlation (ρ = 0.97 using one method and .94 using another method) between ChatGPT's error detection and human coding, although this correlation diminishes with lower proficiency levels. While ChatGPT infrequently misidentifies errors, it often underestimates the total error count. The study also highlights ChatGPT's limitations, such as the issue of consistency, and provides guidelines for future research applications.
Narratives of TESOL professionals: Navigating the doctoral program. Information Age Publishing, 2023
In this chapter, I reflect on my academic journey, with a particular focus on my PhD years, to show how I grappled with native-speakerism as an international multilingual scholar. I went from eagerly seeking to acquire standard, native-like English to accepting my non-native identity and finally reflecting critically on the Janus-faced reality of being torn between the romanticized ideals of multilingualism and the monolingual ideology of standard English. By sharing how I shuttled between idealism and reality in my research and teaching, I elucidate the myriad facets of multilingualism, both the opportunities it brings as well as its concomitant challenges. I argue that the idea of embracing languages and language varieties other than standard English is empowering and liberating for many multilingual scholars, yet such idealized conceptualization often falls short in high-stakes situations where linguistic hierarchy and English monolingual ideology continue to prevail.
FINDING THE RIGHT VOICE(S): AN ENGAGEMENT ANALYSIS OF L2 WRITERS IN HYPOTHETICAL LEGAL WRITING
Linguistics and Education, 2023
This longitudinal case study tracks the development of four second language (L2) writers’ skills in hypothetical legal writing in a year-long legal language program. Drawing on the system of Engagement from systemic functional linguistics, the study analyzes how L2 writers engaged different legal voices and advanced their arguments via three discursive strategies: dialogic expansion, contraction, and justification. An examination of the Engagement resources the writers deployed in 32 essays illustrates their diverse developmental paths and highlights the linguistic choices that reflect the variation in their development. I discuss the influence of initial L2 proficiency and model essays on L2 writers’ trajectories and the distinct challenges these writers faced in maintaining a consistent argumentative position. I argue that the system of Engagement is a useful analytical framework for understanding the linguistic choices L2 legal writers make as they work toward the communicative goals of the target legal genre.
ProQuest Dissertations & Theses Global, 2020
In the past three decades, the construct of second language (L2) writing complexity has been theorized and refined in both second language acquisition (SLA) (Crossley, 2020; Housen, De Clercq, Kuiken, & Vedder, 2019; Lu, 2011; Norris & Ortega, 2009) and Systemic Functional Linguistics (SFL) research (Byrnes, 2009; Ryshina-Pankova, 2015; Schleppegrell, 2004). The general consensus is that lexical and syntactic variations are regarded as signs of advanced academic writing. The contemporary legal writing pedagogy, however, is informed by the Plain English Movement (Benson, 1985; Dorney, 1988; Felsenfeld, 1981), which largely discourages the use of overly complex structures and “elegant variations.” The recommendation to use plain English in legal writing thus poses a challenge to the theoretical consensus in SLA and SFL research and raises a question about the conceptualizations and assessment of writing complexity in academic legal writing classrooms.
This dissertation consists of three interrelated studies and aims to address this contradiction by examining the development and assessment of writing complexity (i.e., lexis, syntax, and discourse) in 246 hypothetical legal essays written by 31 international (LL.M.) students over one year of legal language study at the Georgetown University Law Center. In Study 1, I used a structural, corpus-based approach and tracked the changes of 31 students’ lexical and syntactic complexity at six data collection points over one year and compared the complexity indices with those benchmarked by eight model essays. In Study 2, I offer an in-depth discussion of four students’ distinct developmental trajectories of discourse complexity, which I analyzed through the system of engagement (Martin & White, 2005) from the SFL perspective. Finally, in Study 3, I adopted a mixed-methods approach to investigate two legal instructors’ conceptualizations of writing complexity in the classroom setting.
Results showed that overall students were able to write with significantly more sophisticated words and, in the second semester, significantly shorter sentences, a pattern consistent with the pedagogical focus of the program. Additionally, the four individual trajectories of discourse complexity indicate that, even when starting with a similar proficiency level, some students constructed increasingly complex legal discourse by actively engaging different legal voices through dialogic expansion and contraction, while others only made marginal progress. Consistent with the instructors’ conceptualizations of writing complexity, structurally more complex essays were not necessarily of higher text quality. In fact, the instructors’ assessment of text quality was found to be influenced by a number of external factors, such as students’ performance in comparison to others and their academic progress in the program. I conclude by discussing additional insights that emerged from the dissertation, such as the boundaries between text modeling and plagiarism, as well as the role of text length in assessing timed writing. Finally, I call for a critical reflection on the pedagogical principles of Plain English and highlight the value of an integrated structural-functional approach to holistically understand the construct of L2 writing complexity in academic legal writing.
Despite the growing interest in interlanguage complexity development in study abroad (SA) research, no clear-cut conclusions can be made as to whether and to what extent learners' interlanguage complexity increases following a sojourn abroad. The current study meta-analyzed the overall effects of study abroad on measured oral and written complexity, as well as the moderator effects (i.e., learner demographics, SA contextual features, and outcome measures) on the variability of interlanguage complexity effect sizes (Cohen's d). A comprehensive search was conducted to obtain studies that have quantitatively documented lexical and syntactic complexity changes during SA through a pre-and-post SA design. A total of 30 independent samples from 28 primary studies involving 602 participants were retrieved and coded for gains and for moderator variables. Results show an overall small effect of study abroad on language complexity development (d = 0.37). In addition, moderator analyses suggest that larger effects are associated with (a) learners at an intermediate proficiency level, (b) learners enrolled in a language study program while SA, (c) programs that implemented a language pledge, or (d) programs with Mandarin Chinese as the target language. More fine-grained and systematic reporting practices are proposed for future research.
RECASTS IN SCMC: REPLICATING AND EXTENDING GURZYNSKI-WEISS ET AL. (2016)
The Routledge handbook of second language research in classroom learning, Routledge, 2019
The present chapter reports on an empirical study that replicates and extends an investigation by Gurzynski-Weiss, Al Khalil, Baralt, and Leow (2016), which addressed the strands of textual enhancement (enhanced vs. unenhanced), learner awareness (low vs. high), and type of linguistic item (lexis vs. morphology vs. syntax) in synchronous computer-mediated communication (SCMC) via the use of recasts and think-alouds. Low-intermediate adult L2 learners of Spanish completed three story retell tasks via iChat with an interlocutor while thinking out loud. Interlocutors immediately provided either an enhanced or unenhanced recast for each target error. The present study employs the same research design and participant population with 40 adult L2 learners of Spanish, while additionally addressing immediate and delayed learning outcomes. The findings indicate that learners tend to be more aware of recasts targeting lexis than grammar. Furthermore, recasts, whether enhanced or unenhanced in SCMC, aid in L2 development, particularly of lexis.
Language and Education, 2019