Attached is the Qrels file for Trec-LiveQA 2015, (to be published on the Trec site for active participants in a few days).
The Qrels format is as follows:
For each question, the first line provides the question info (tab separated) and the rest of lines (with the same trec-id) keep all answers for that question:
trec-id qid title body main-category sub-category
trec-id qid answer-score answer resource
trec-id qid answer-score answer resource
....
A few comments:
1. Answer scores are:
-2: non-readable
1: poor
2: fair
3: good
4: excellent
2. Q&A page content can be scraped from the Yahoo Answers using the given scrapping program: https://github.com/yuvalpinter/LiveQAServerDemo/blob/master/src/org/trec/liveqa/GetYAnswersPropertiesFromQid.java
3. Answers are provided as is, without the participant name. However, some participants added their system name to the answer. We tried to strip out these tags however we might had missed some. Please let us know if you find such tags.
Thanks