This is a companion page to the paper Inter-coder agreement for computational linguistics, a survey of coefficients of agreement (such as Kappa and Alpha) that appear in Computational Linguistics in 2008.
Whereas Kappa is reasonably well-known and software for computing it can be easily found and is incorporated in many statistical packages, Alpha is less understood and more difficult to use. The aim of this page is to keep track of resources concerning alpha, such as introductory material, software, and papers using Alpha for computational linguistics.
The main source of documentation on Alpha is Klaus Krippendorff’s Content Analysis textbook – we recommend the 2nd edition as it contains a much more extensive discussion of Alpha:
Klaus Krippendorff. Content Analysis, An Introduction to Its Methodology 2nd Edition; 413 pages. Thousand Oaks, CA: Sage Publications, 2004.
Krippendorff also wrote a paper specifically on the differences between Alpha and Kappa:
Klaus Krippendorff (2008). Systematic and Random Disagreement and the Reliability of Nominal Data. Communication Methods and Measures, 2(4), 323–338.
Further information and a number of useful practical manuals can be found in Klaus Krippendorff’s page. In particular, we would suggest looking at http://www.asc.upenn.edu/usr/krippendorff/webreliability.doc
For the use of Alpha in Computational Linguistics, apart from our paper, we recommend having a look at Passonneau’s LREC 2004 paper.
From Klaus Krippendorff’s page
Krippendorff has scripts for computing alpha but we are not sure about their availaibility
See http://www.asc.upenn.edu/usr/krippendorff/webreliability.doc
Andrew Hayes and Klaus Krippendorff’s SPSS macros
Available from http://www.comm.ohio-state.edu/ahayes/
Select “SPSS and SAS Macros” then KALPHA.SPS
the package is described at www.asc.upenn.edu/usr/krippendorff/CMM_alpha_final.doc
Thomas Lippincott’s and Becky Passonneau’s Python scripts
Thomas Lippincott and Becky Passonneau developed a Python script that can be used to compute a variety of agreement coefficients.
Ron Artstein’s Alpha Calculation script
Ron Artstein wrote several scripts to compute alpha for the experiments ran in the ARRAU project. This script is a version for general use, computing interval alpha.
Another SAS macro for computing Alpha
Computing Alpha in R
A general site on software for computing reliability
The excellent site http://www.temple.edu/sct/mmc/reliability/ reviews the available software for computing reliability
Ron Artstein and Massimo Poesio, 2008.
Inter-coder agreement for computational linguistics. (PDF, 41 pages)
Computational Linguistics, 34(4), 555-596.
Extended version (PDF, 66 pages) .
Passonneau, Rebecca; Lippincott, Tom; Yano, Tae; Klavans, Judith. 2008. Relation between Agreement Measures on Human Labeling and Machine Learning Performance Results from an Art History Domain. In Proc. of Language Resources and Evaluation (LREC). pdf
Massimo Poesio and Ron Artstein.
Anaphoric annotation in the ARRAU corpus. (PDF)
LREC 2008, Marrakech, Morocco, May 2008.
Ron Artstein and Massimo Poesio.
The Arrau corpus of anaphoric relations. (PDF slides)
American Association for Corpus Linguistics, Provo, Utah, March 2008.
Nenkova, Ani; Passonneau, Rebecca J.; McKeown, Kathleen. 2007. The pyramid method: incorporating human content selection variation in summarization evaluation. ACM Transactions on Speech and Language Processing, 4(2). pdf
Passonneau, Rebecca J. 2006. Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In Proceedings of the Fifth Internation al Conference on Language Resources and Evaluation (LREC). pdf
Ron Artstein and Massimo Poesio.
Identifying reference to abstract objects in dialogue. (PDF)
brandial 2006 proceedings, Potsdam, Germany, September 2006.
Ron Artstein and Massimo Poesio.
Bias decreases in proportion to the number of annotators. (PDF)
Proceedings of FG-MoL 2005, pages 141-150. Edinburgh, Scotland, August 2005.
Massimo Poesio and Ron Artstein.
Annotating (anaphoric) ambiguity. (PDF)
Corpus linguistics, Birmingham, England, July 2005.
Massimo Poesio and Ron Artstein.
The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. (PDF)
Proceedings of the Workshop on Frontiers in Corpus Annotation II: Pie in the Sky, pages 76-83. Ann Arbor, Michigan, June 2005.
Ron Artstein and Massimo Poesio.
Kappa3 = alpha (or beta).
Technical Report CSM-437, University of Essex Department of Computer Science, September 2005.
Superseded by journal article Inter-coder agreement for computational linguistics, above.
Nenkova, Ani and Rebecca J. Passonneau. 2004. Evaluating Content Selection in Summarization: the Pyramid Method. NAACL. pdf
Passonneau, Rebecca J. 2004. Computing Reliability for Coreference Annotation. In: Proceedings of Language Resources and Evaluation (LREC). pdf
We list here some useful papers and / or websites on coefficients of agreement not necessarily focusing on alpha
Jean Carletta, (1996). Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics, 22(2), 249-254. PDF.
Ron Artstein and Massimo Poesio.
Inter-coder agreement for computational linguistics. (PDF, 41 pages)
Survey article, to appear in Computational Linguistics.Extended version (PDF, 66 pages) .
John Uebersax’s page, and in particular http://ourworld.compuserve.com/homepages/jsuebersax/agree.htm
The ARRAU project was funded by the EPSRC, grant number GR/S76434/01