9. Above board: Results of the explicit use of ChatGPT and other online tools  in foreign language translation coursework within a Higher Education context  

Authors:  Enza Siciliano Verruccio and Cameron Powell

Institution: University of Reading

Situation 

This case study presents an evaluation of the introduction of ChatGPT to the assessment of a 10-hour translation course for final-year Modern Languages undergraduates studying Italian at the University of Reading (UK). Until 2020 the course was assessed through an in-person final examination with no access to dictionaries. In 2021, following the move to online assessment prompted by COVID, and its perceived potential for academic misconduct (Holden, Norris and Kuhlmeier, 2021), translation into English became a coursework project which introduced set annotations alongside the translation output to test students’ explicit linguistic competence.

Task 

The new project format was intended to address, albeit partially, major concerns within the discipline regarding the robustness and validity of language assessment in non-exam conditions, but without explicitly attending to students’ use – or abuse – of web-based machine translators (Paterson, 2023). Within this context, the emergence of ChatGPT appeared as the ultimate threat to the survival of language coursework (Cotton, Cotton and Shipway, 2024), which led to question the adequacy of the approach taken (Rudolph, Tan and Tan, 2023) and prompted further intervention.  

Alongside preserving the robustness of testing and academic integrity, this new intervention aimed also to promote an informed and critical use of AI-powered tools and thus restore language-learner agency (Muñoz-Basols et al., 2023). Also important was increasing students’ digital capabilities and strengthening their employability prospects (Jisc, 2023).  

Action 

In order to bring the use of online machine translators ‘above board’, not only did the remit of the translation project need to be enhanced and expanded, but the course itself needed to be redesigned and its focus fully shifted from product to process.  Students were therefore exposed from the start to source texts presented alongside translations provided in the first instance by Google Translate and DeepL and, from 2023-24, also by ChatGPT. Students – effectively transformed into post-editors (Niño, 2008) – were asked to compare and comment on the machine translations before producing their own versions (Ducar and Schocket, 2018; Pardo-Ballester, 2022). This format was reproduced in the coursework assessment where, in 2023-24 ChatGPT was also used to produce an answer to the most complex annotation, which students had to evaluate (Figure 1). 

Figure 1: ChatGPT answer to Annotation 4, Translation into English Project 2023-24

Particular attention was given to the choice of source texts. For the project, this included multilayered linguistic modes and registers, expletives, pragmatic markers, and cultural references which, besides proving challenging for AI translators, would maximise the necessary student engagement with the AI output.  

The assessment design was guided by considerations of accessibility and ease of navigation. Being a novel format, the brief was detailed, acting as a table of contents with internal hyperlinks (Figure 2). The source text and the three AI versions were colour-coded and placed in adjacent columns. Opportunities for more targeted comments and for reflection on the translation process were added. With the change in learning outcomes and task focus, marking criteria also needed to be fully reviewed.

Figure 2: The brief to the Translation into English Project 2023-24

Results 

Within an institutional context of highly harmonised assessment approaches across languages and of a reluctance to abandon traditional formats, the intervention in the Italian into English translation project constitutes a true innovation and the first attempt to engage openly with students’ use of AI tools in language teaching at Reading. As such, the results of the intervention were declared impactful by the university’s centre for teaching development and are going to be disseminated to a school-wide event on AI and assessment in the Humanities.

On reviewing student performance, the most positive impact appeared to be on the level of engagement with the task, as all students showed clear attempts at deploying a critical approach and high-end linguistic skills. The marks displayed an increased spread towards lower categories, which had virtually disappeared from students’ results in the coursework from previous years. High-achieving students still performed extremely well, and students with learning differences did not appear to be adversely affected. This outcome puts this assessment format on a par with the last in-person exam (2019) held for this course in terms of robustness, thus confirming it as a valid alternative to the call for a return to exams (Susnjak, 2022).

The project is also highly scalable and sustainable, and easily adaptable to other languages, across institutions and levels of language education. The initial increase in time and resource input is offset by its approach and design being more resilient to fast-paced changes in technology. 

Stakeholder Commentary

The success of the project is also confirmed by students, who in their feedback reported that the task allowed them to assess and acknowledge the limitations of the online tools and to assert their agency (‘regaining power’ and ‘control’ were the words most used) by forcing them to engage critically with, but also go beyond, the need to use online translators. In comparison to translation projects in other languages, the task felt ‘engaging’, ‘authentic’ and ‘allowed [them] to treat translation as [they] would professionally.’

Reported instances of task frustration and suggestions on the layout on the basis of the large quantity of information presented will inform future iterations of the project to improve its accessibility.

References

Cotton, D. R. E., Cotton, P. A. and Shipway, J. R. (2024) ‘Chatting and cheating: Ensuring academic integrity in the era of ChatGPT’, Innovations in Education and Teaching International, 61(2), pp. 228–239. Available at:  https://doi.org/10.1080/14703297.2023.2190148

Ducar, C. and Schocket, D.H. (2018) ‘Machine Translation and the L2 Classroom: Pedagogical Solutions for Making Peace with Google Translate’, Foreign Language Annals, 51(4), pp. 779–795. Available at: https://doi.org/10.1111/flan.12366

Holden O. L., Norris M. E., Kuhlmeier V. A. (2021) ‘Academic Integrity in Online Assessment: A Research Review’, Frontiers in Education, 6. Available at:   https://doi.org/10.3389/feduc.2021.639814

Jisc (2023) Designing Learning and Assessment in a Digital Age. Available at: https://www.jisc.ac.uk/guides/designing-learning-and-assessment-in-a-digital-age

Accessed on 8/4/24.

Muñoz-Basols, J., Neville, C., Lafford, B.A., & Godev, C. (2023) ‘Potentialities of Applied Translation for Language Learning in the Era of Artificial Intelligence’, Hispania, 106(2), pp. 171-194. Available at: https://doi.org/10.1353/hpn.2023.a899427. 

Niño, A. (2008) ‘Evaluating the Use of Machine Translation Post-Editing in the Foreign Language Class’, Computer Assisted Language Learning: An International Journal, 21(1), pp. 29–49. Available at: https://doi.org/10.1080/09588220701865482

Pardo-Ballester, C. (2022) ‘A Case Study: A Technological Insight on Teaching Translation’, Foreign Language Annals, 55(3), pp. 894–913. Available at: https://doi.org/10.1111/flan.12640

Paterson, K. (2023) ‘Machine translation in higher education: Perceptions, policy, and pedagogy’, TESOL Journal, 14(2). Available at: https://doi.org/10.1002/tesj.690. 

Rudolph, J., Tan S., Tan Sh. (2023) ‘ChatGPT: Bullshit Spewer or the End of Traditional Assessments in Higher Education?’, Journal of Applied Learning and Teaching, 6(1), pp. 342-363. Available at: https://doi.org/10.37074/jalt.2023.6.1.9

Susnjak, T. (2022) ‘ChatGPT: The End of Online Exam Integrity?’,  arXiv. Available at: https://doi.org/10.48550/arXiv.2212.09292

Author biographies

Enza Siciliano Verruccio 

Enza Siciliano Verruccio is Associate Professor in Second Language Education at the University of Reading, where she teaches Italian language and translation. Her most recent research focus is the use of AI tools in foreign language learning and teaching.

Cameron Powell

Cameron Powell is a final-year French and Italian student at the University of Reading. He is interested in the use of AI in translation and how it will impact the future of foreign language study.