Chapter 2. Coefficients of Agreement and Chapter 3. Using Agreement Measures for CL Annotation Tasks
A list of resources for these chapters can be found here.
Chapter 5. Probabilistic Models of Annotation
The Recognising Textual Entailment (RTE) dataset used as a running example in this Chapter can be downloaded here.
An implementation of the Multi-Annotator Competence Estimation (MACE) model described in Section 5.3 Modeling Annotator Behaviour can be found here.
An implementation of the community model described in Section 5.5 Hierarchical Structures can be found here.
The models and data used in the experiments for Section 5.7 Modeling Sequence Labelling Tasks can be found here.
The implementation of the Mention Pair Annotation (MPA) model described in Section 5.8 Aggregating Anaphoric Annotations can be found here. The Phrase Detectives coreference corpus used in the experiments can be downloaded here.
An implementation of the variational autoencoder models and the data used in Section 5.9 Aggregation with Variational Autoencoders can be found here.
An implementation of the methods described in Section 5.10 Modeling Complex Annotations can be found here.