2011 TREC Crowdsourcing Track - TREC Crowdsourcing Track

Looking for information on the 2012 track? Go here.

2011 Track

Other Years

Organizers

Gabriella Kazai, Microsoft Research
Matthew Lease, University of Texas at Austin

Information

Overview paper (forthcoming...)
Track Guidelines
Overview slides (from TREC 2011)
Initial Track Proposal
Track Proposal Slides (from TREC 2010)
Participant Slides (from TREC 2011)

Practical and Effective Design of a Crowdsourcing Task for Unconventional Relevance Judging: The University Carlos III of Madrid at TREC 2011 Crowdsourcing Track by Julian Urbano

Data

Task 1: Assessment

Crowd judgments submitted by participating teams (worker IDs have been anonymized)
Test topics and Data
Development Data

ClueWeb data

If you **have not** already signed and returned the ClueWeb agreement

Go to the clueweb site: http://lemurproject.org/clueweb09/index.php#Obtaining
Fill out and sign the permission PDF, being sure to check the box for the "TREC 2011 Crowdsourcing subset"
Return the form (by fax or email) following their instructions

If you **have** already signed the ClueWeb agreement

Email Dana (dhouston at cs dot cmu dot edu) with subject "request for TREC 2011 Crowdsourcing subset". Be sure to identify yourself and your organization in sufficient detail that Dana can match both to your already submitted ClueWeb agreement.

Download here (requires username / password provided by CMU)

Task 2: Aggregation

Test Set (August 19, 2011)
Development Set

Evaluation Software (by Hyun Joon Jung, University of Texas at Austin)

Submissions and Results (restricted to participating teams)

Schedule

August 8: Release of Task 1 test data
August 19: Release of Task 2 test data
September 15: Submissions due
October 3: Preliminary results announced to participants
October 24: Team papers due to NIST for inclusion in TREC conference notebook
Nov 15-18: Text REtrieval Conference (TREC 2011) at NIST campus in Gaithersburg, MD
January 25, 2012: Final team papers due to NIST for inclusion in TREC conference proceedings

Track Sponsors

Thanks to our sponsors for helping make the 2011 track a great success!

CrowdFlower (see their 4/26/11 blog post): $100 credit per participant + support. CrowdFlower will provide technical support and resources to assist teams with an introduction to the CrowdFlower self-service platform. Email Joseph Childress (joseph at crowdflower dot com) with Subject "TREC 2011 Crowdsourcing Track credit request" to receive your credit and inquire regarding technical support.

Amazon: $300 reimbursement per team using Mechanical Turk in submitted results