PRELEARN (Prerequisite Relation Learning) is a shared task on concept prerequisite learning which consists of classifying prerequisite relations between pairs of concepts distinguishing between prerequisite pairs and non-prerequisite pairs. For the purposes of this task, prerequisite relation learning is proposed as a problem of binary classification between two distinct concepts (i.e. a concept pair).

In other words, the goal of the task can be defined as follows:

In the present task, concepts are single or multi word domain terms corresponding to the title of a page on the Italian Wikipedia. For example, Prodotto scalare and Aritmetica are both concepts of the Precalculus domain, but the two concepts are also the titles of two Italian Wikipedia pages (se Prodotto scalare and Aritmetica on Wikipedia). In this scenario, identifying a prerequisite relation can be understood as understanding if the content of a Wikipedia page contains the prior knowledge required to understand the content of another Wikipedia page (i.e. the target page).

We ask PRELEARN participants to achieve this goal in at least one of the task settings described below.

Task Settings

We define four sub-tasks for addressing automatic concepts prerequisite learning: two of them are concerned with the model that participants can use for tackling the task, the other two distinguish different classification scenarios where the proposed model can be tested.

We ask participants to submit at least one model tested on both scenarios.

Models

For what concerns the model of the submitted system(s), we define the following two settings based on the features used for training:

• a model that acquires information only from raw text (e.g. Wikipedia pages, corpora for distributional representations);

• a model that can rely both on raw text and structured information (e.g. Wikipedia graph and metadata, DBpedia, organization in sections and paragraphs, etc.).

In any case, we recommend not to use any type of prerequisite-labelled data (apart from the training dataset released by PRELEARN organizers).

Scenarios

The implemented model(s) can be tested in the following scenarios:

• train the model(s) on any domain (in-domain setting);

• train the model(s) on anything but the domain of the test set (cross-domain setting).

Examples

To better understand the task settings that we propose, we show a classification of four works on automatic prerequisite learning published in the past years according to the four settings of the present shared task.

These are the works we consider as examples:

• CrowdComp (Talukdar and Cohen 2012): this paper presents a Maximum Entropy classifier to predict prerequisite relations between Wikipedia pages. The classifier exploits three types of features: Wikipedia hyperlinks features (i.e. random walk with restart (RWR) score between two pages and PageRank score) edits of pages (i.e. RWR score on edit information), page content (i.e. category identity of a page, presence of a link, mention of a concept in the text). Most of these features can't be acquired from the sole textual content of a Wikipedia page, thus we classify this model as a structured-based model. Tests are performed both with a in-domain and a cross- domain training.

• Ref-D (Liang et al. 2015): the reference distance metric is a metric for prerequisite relation identification which computes the associative strength of two concepts. The formula for computing Ref-D is quite generic: the authors claim that the importance of a concept for another can be computed based on links between the Wikipedia pages of the concepts, the mentions in books, the number of citations in scientific papers, etc. The experiments described in the paper present a Wikipedia-based Ref-D implementation exploiting hyperlinks between the Wikipedia pages of the concepts. For this reason, we consider this method as a structured-based model.

• Burst (Adorni et al. 2019): this work proposes a method based on Burst Analysis to identify relevant portions of texts (burst intervals) for each concept and than exploits temporal patterns between burst intervals to find concept pairs showing a prerequisite relation. Since burst intervals are computed based on concept occurrences in the text and that relations are found based on manually defined patterns, this method exploits only raw text information. The system was tested on the content of a textbook on Computer Science.

• Ita-Prereq (Miaschi et al. 2019): Ita-Prereq model proposes to exploit a deep learning strategy to address automatic prerequisite learning. The features used by the classification model are both lexical features (i.e. word embeddings) and global features extracted from the text of Wikipedia pages (e.g. the presence of the prerequisite concept in the content of the target concept page, the Jaccard similarity between two pages, the length of the pages in terms of tokens, etc.). The Ref-D metric used as global feature in this system, rather than being implemented using Wikipedia hyperlinks, exploits concept mentions. Any information coming from Wikipedia structure (e.g. hyperlinks, category, metadata) or from other external resources were used. The model was tested both in a in-domain and cross-domain scenario.

Look at the table below to have a summary of how we classified the models described above.