The Task

The goal of the CRAFT Coreference Resolution task (CRAFT-CR) is to automatically reproduce identity chains in the documents of the CRAFT corpus. The challenges specific to this coreference task lie in the fact that identity chains can span the length entire documents, and that the base noun phrase mentions linked in these chains are permitted to have discontinuous spans (see example below).

Description of Data

Development Data

The training data for this task consist of the already publicly released identity chain annotations in the 67 articles of the CRAFT Corpus v3.1.3.

Evaluation Data

The testing data for this task consist of identity chains in 30 articles of the corpus that have not yet been publicly released. These articles have been annotated in exactly the same ways as the publicly released document set.

Evaluation Metrics

Evaluation of the CRAFT-CR task will an adaption of the evaluation code used for the CoNLL Coref 2011/12 shared tasks. The original evaluation code has been modified in two ways:

1. The code has been modified to handle mentions with discontinuous spans. The original code required mentions to have only contiguous spans.

2. The code has been modified to allow for partial mention matches, i.e. to allow mentions that overlap to match.

Coreference resolution system performance will be evaluated on all 6 metrics provided:

• muc: MUCScorer (Vilain et al, 1995)
• bcub: B-Cubed (Bagga and Baldwin, 1998)
• ceafm: CEAF (Luo et al., 2005) using mention-based similarity
• ceafe: CEAF (Luo et al., 2005) using entity-based similarity
• blanc: BLANC (Luo et al., 2014) BLANC metric for gold and predicted mentions
• lea: LEA (Moosavi and Strube, 2016) Link-based Entity Aware Metric

The evaluation platform is now available in the CRAFT Shared Task GitHub repository. See the wiki for installation and usage instructions.

Input/Output

For final submissions, users will be provided tokenized versions each document. The evaluation script cannot handle differences in tokenization, thus gold-standard tokens will be provided as input to your system. Input and output files for the CRAFT-CR task will both use the CoNLL Coref 2011/12 file format, with a modification to allow for mentions with discontinuous spans.

Discontinuous mentions are denoted by the addition of a character or characters (non-digit) after the chain identifier (integer). In the example below, all spans denoted by 32a belong to the same discontinuous mention that is a member of identity chain #32. The next discontinuous mention in identity chain #32 (if there is one), would be denoted by 32b. There is no constraint on the number of characters that can be used to identify a mention, e.g. 32aaaaaaa is perfectly valid. The only constraint is that the only integer in the mention identifier must apply to the identity chain, and that the integer must occur at the beginning of the identifier string, e.g. aaaaaa32 is not a valid mention identifier, nor is 32abc123.

Example of the CoNLL Coref 2011/12 file format with discontinuous span handling

There are three mentions in the sentence below extracted from document PMC194730 (PMID: 12925238). Each mention is a member of a different identity chain. Coreference information is located in the final column of this data file format.

1. "The protein"

• a contiguous mention that starts at token 1 and ends at token 2 (inclusive)
• a member of identity chain #2

2. "a variable N-terminal .. domain"

• a discontinuous mention that includes tokens [16-20] and token 27
• a member of identity chain #32

3. "a conserved C-terminal domain"

• a contiguous mention that includes tokens [22-27]
• a member of identity chain #33