This task, organised in 2008, required participants to develop a method which, given an ATTRIBUTE-SET representing the semantic content of an identifying description for a target entity, outputs a WORD-STRING, that is, a realisation in English of the semantic representation.
The training and development data consisted of the same TUNA corpus instances used for TUNA-AS and TUNA-REG. A new test set, consisting of 112 corpus instances, was developed for TUNA-R (only), using the same methodology as that used for the original TUNA corpus collection.
The same software used for TUNA-AS and TUNA-REG was provided for this task. For WORD-STRINGs that are the output of the TUNA-R and TUNA-REG tasks, the software compares human and peer outputs on the basis of (i) string edit (Levenshtein) distance; (ii) string accuracy, that is, the proportion of peer output strings that were identical to human-produced descriptions.
Other metrics: In addition to edit distance and string accuracy, we also computed BLEU (Papineni et al., 2002) and NIST n-gram similarity scores comparing the peer and human outputs over the test entire test sets.
The above software can be found in the TUNA'08 participants' pack: [TUNA'08 pack]. Descriptions of the human evaluation methods can be found in the TUNA'08 results report (see below).
Detailed documentation for the TUNA-R shared task can be found in the TUNA'08 participants' pack: [TUNA'08 pack].
W08-1131: Albert Gatt; Anja Belz; Eric Kow
The TUNA Challenge 2008: Overview and Evaluation Results
W08-1132: Bernd Bohnet
The Fingerprint of Human Referring Expressions and their Surface Realization with Graph Transducers (IS-FP, IS-GT, IS-FP-GT)
W08-1133: Giuseppe Di Fabbrizio; Amanda J. Stent; Srinivas Bangalore
Referring Expression Generation Using Speaker-based Attribute Selection and Trainable Realization (ATTR)
W08-1134: Pablo Gervás; Raquel Hervás; Carlos León
NIL-UCM: Most-Frequent-Value-First Attribute Selection and Best-Scoring-Choice Realization
W08-1136: John D. Kelleher; Brian Mac Namee
Referring Expression Generation Challenge 2008 DIT System Descriptions (DIT-FBI, DIT-TVAS, DIT-CBSR, DIT-RBR, DIT-FBI-CBSR, DIT-TVAS-RBR)