To measure writing proficiency, we asked learners to spend approximately 25 minutes writing—under exam conditions (i.e., without a translation software or a dictionary to write—at least 100 words about a book or a film that they had recently read or watched.
Learners' writing samples were first transcribed and then tokenised, tagged, lemmatised, and dependency-parsed using the natural language processing package, UDPipe, in R.
UDPipe is an R package that provides a simple and efficient interface for tokenisation, lemmatisation, part-of-speech tagging, and syntactic parsing using Universal Dependencies (UD) models. Universal Dependencies is a framework for consistent annotation of grammar across different languages, and UDPipe implements pre-trained models for these annotations.
From this output, we then computed proficiency-appropriate measures.
In addition to analysing the linguistic features of learners' writing samples, we also asked trained research assistants to rate human writing samples in terms of content, organisation, accuracy of vocabulary and grammar, and range of vocabulary and grammar. The full marking criteria can be found here.