TREC 2015
As part of the National Institute of Standards and Technology (NIST) Text REtrieval Conference (TREC), the Contextual Suggestion Track deals with complex information needs which are highly dependent on context and user interests.
Mailing list: https://groups.google.com/forum/#!forum/treccontext
Twitter: @treccontext
Register for TREC in order to participate in this track: http://trec.nist.gov/pubs/call2015.html
Slides of the presentations at TREC'15.
Track Organizers
Charles L. A. Clarke, University of Waterloo
Adriel Dean-Hall, University of Waterloo
Jaap Kamps, University of Amsterdam
Julia Kiseleva, Eindhoven University of Technology
Live Experiment Quick Start
The guidelines have more details about the track but if you want to quickly get started with the track here's how:
Create a web server that can respond to:
curl -H "Content-Type: application/json" --data @request.json -XPOST http://YOUR_SERVER
The request.json file has this JSON schema: https://github.com/akdh/cst-tools/blob/master/request_schema.json and responds must be in this JSON schema (https://github.com/akdh/cst-tools/blob/master/request_schema.json).
An example of a request and a response can be seen here: https://gist.github.com/akdh/564cbb5f9c6c92c99d8a
The response contains an ordered list of attraction IDs from this collection: http://plg.uwaterloo.ca/~adeanhal/trec2015/collection_2015.csv.gz. These attraction IDs should be suggestions that the user represented in the request would like.
Once your server is set up register it here: https://docs.google.com/forms/d/1a1kxj_EPvyOmB3Mx7YK3wOz1UQ_rUx8sbhuMTWGxR00/viewform?usp=send_form#start=invite and we will start sending requests to it!
Batch Experiment Quick Start
In order to submit a run for the batch experiments generate a file that has all the responses to the requests in batch_requests.json (see below). Before submitting (on the TREC website) make sure your generated responses file is valid by running batch_validate.py (replace RUN_FILE with your filename):
python batch_validate.py batch_requests.json RUN_FILE
Both batch_requests.json and your run file contain a list of JSON objects separated by newlines. Each request is:
{
"body": {...},
"candidates": [...],
"id": 2
}
where body contains the request in the same format as the live experiment, candidates is a list of document IDs that are valid to return, and ID is the request id. Each response should be:
{
"body": {
"suggestions": [...]
},
"groupid": "demo",
"id": 2,
"runid": "demoA"
}
where the body contains the response in the same format as the live experiment, which is just a list of suggestions, each item in the list is a document ID from the candidates list in the request. Use your groupid and runid in place of "demo" and "demoA" and the id is the request ID that this response corresponds to. A sample run file is available below (sample_batch_response.json).
Contexts
The contexts (target cities) for 2015 are available for download below. Lat/long coordinates have been provided in a separate file for convenience, these do not represent the exact position of the user but rather are analogous to the name of the city.