Shared Annotation Task

The annotation task will call for identifying instances of event mentions and coreference links, including bridging references, with an optional layer of post-coreference inference.  A small number of texts will be distributed to participants to be annotated.  Each team will use their own guidelines for this annotation, which may or may not be domain dependent, and may or may not include bridging references.  If teams choose to limit the events they include, this will require an explanation of why based on a domain event list/hierarchy for the data.  Teams also have to agree to share their guidelines with other participants.

Teams who already have agreed to submit data annotations include:

·         LDC                          ACE annotation

·         CUNY/UT-Dallas      ACE annotation

·         CMU                         event coreference, subevent, and membership coreference

·         Colorado                   event coreference, type/subtype coreference and membership

·         Stanford/Col/LDC     event coreference

·         BBN                          OntoNotes coreference w/ semantic relations and implicit arguments

 

Any additional groups who would like to provide annotation using new or existing guidelines are welcome to contact: Teruko Mitamura teruko@cs.cmu.edu.

You can download a sample document from here.
 
Annotated Texts Submission:

In order to submit the annotated text in EasyChair, choose Shared Annotation Task in Events-2013.  

Then submit your annotated results in zip format (preferred) or txt format at:

 https://www.easychair.org/conferences/submission_new.cgi

 
When you submit, please indicate 1st Round, 2nd Round and 3rd Round.
 
Annotation Instruction:
 
There are two parts for annotation. 
 
Part 1: Markables
 
The purpose of this part is to identify the Markables. A Markable is a mention of an event (typically, a verb or eventide noun) that should be annotated according to your scheme. If the mention includes more than one word, please pick the main word of the expression. Each Markable mention has a unique identifying number. 

To annotate a Markable, please insert before the mentioning word
  <MARK  ID=n>
and after it
  </MARK>.
Make sure each n is an integer unique in that text.  

If you wish to associate additional information with the Markable (for example, that it is a Modality indicator), please make a label and a value according to your annotation scheme and add it into the initial Markable symbol, viz.
  <MARK  ID=n  MODAL=T>

Example:
  "He  <MARK ID=1 MODAL=T>may</MARK>  never  <MARK ID=2>call</MARK>  her  up."

Note that "up" is not marked, even though we know it is part of the event. 
 
Part 2: Coreference
 
The purpose of this part is to relate the various markables according to your theory of coreference linkage. Each link is a triple, with possible additional information about that link. For every two Markables that you consider coreferent (or otherwise linked), please add one triple, on a separate line, to the bottom of the file, using the format
 
RELATION Markable1-ID Markable2-ID
 
where the two Markables are represented by their ID numbers, as in
 
coref-identical 1 5
 
If you wish to add more information about the relation, for example that it is only hypothetical, (if that is what your theory calls for), then please make up a suitable notation and add the information after the triple on the same line. You can add more than one piece of information, as in
 
coref-identical 1 5 hypothetical certainty=.5
 
that for example might signal that mention 1 is coreferent with mention 5 but that it is only a hypothetical case, and that the annotator's certainty was 0.5. If the information your theory of coreference calls for cannot be fitted into this annotation scheme, please contact us.