In LangLearn, the process of language development (LD) is cast as a binary classification problem and it consists in predicting the relative order of two essays written by the same student. Specifically, we follow the approach devised in (Miaschi et al., 2021): given a pair of essays written by the same student (Essay 1, Essay 2), we ask the participants to predict the correct order of the two essays.
In other words, the goal of the task can be defined as follows:
Given a randomly ordered pair (Essay 1, Essay 2) written by the same student, we ask to predict whether or not document e1 was written before document e2.
For the LangLearn task, we rely on two corpora: CItA and COWS-L2H. Please refer to the Data page for their description.
We ask the participants to predict the order of pairs of essays extracted from the CItA and COWSL2H datasets. More specifically, we define two sub-tasks, based on the resources that can be used for training the models:
Sub-task 1: Prediction of language development using ONLY the training data released for the task, i.e. the essays from the CItA or COWS-L2H corpus, or from both corpora
Sub-task 2: Prediction of language development using information acquired from the training data released for the task and also from additional external resources.
Each test set will be evaluated independently from the others by using standard metrics, such as Accuracy (A), Precision (P), Recall (R) and F1-score (F1).
The measures will be evaluated per corpus and the final ranking will be based on the global ranking of each participant, calculated by averaging the macro F1-score from the two corpora.