Looking for information on the 2012 track?  Go here.

2011 Track

Other Years
Organizers Information

Task 1: Assessment

Crowd judgments submitted by participating teams (worker IDs have been anonymized)
Test topics and Data
Development Data 
ClueWeb data
    • If you **have** already signed the ClueWeb agreement
      • Email Dana (dhouston at cs dot cmu dot edu) with subject "request for TREC 2011 Crowdsourcing subset". Be sure to identify yourself and your organization in sufficient detail that Dana can match both to your already submitted ClueWeb agreement.
    • Download here (requires username / password provided by CMU)
Task 2: Aggregation

Test Set (August 19, 2011)
Development Set

Evaluation Software (by Hyun Joon Jung, University of Texas at Austin)

Submissions and Results (restricted to participating teams)

  • August 8: Release of Task 1 test data
  • August 19: Release of Task 2 test data
  • September 15: Submissions due
  • October 3: Preliminary results announced to participants
  • October 24: Team papers due to NIST for inclusion in TREC conference notebook
  • Nov 15-18: Text REtrieval Conference (TREC 2011) at NIST campus in Gaithersburg, MD
  • January 25, 2012: Final team papers due to NIST for inclusion in TREC conference proceedings

Track Sponsors

Thanks to our sponsors for helping make the 2011 track a great success!

CrowdFlower (see their 4/26/11 blog post): $100 credit per participant + support. CrowdFlower will provide technical support and resources to assist teams with an introduction to the CrowdFlower self-service platform. Email Joseph Childress (joseph at crowdflower dot com) with Subject "TREC 2011 Crowdsourcing Track credit request" to receive your credit and inquire regarding technical support.

Amazon: $300 reimbursement per team using Mechanical Turk in submitted results

Matt Lease,
Aug 30, 2011, 12:23 PM
Matt Lease,
Apr 11, 2011, 7:34 AM
Matt Lease,
Jun 13, 2011, 5:31 PM
Matt Lease,
Jun 13, 2011, 6:22 AM
Matt Lease,
Aug 9, 2011, 3:58 AM
Matt Lease,
Aug 17, 2011, 3:16 AM
Matt Lease,
Aug 22, 2011, 5:27 PM