The evaluation script takes in, as input, a submission file, the topics.csv file, and all three qrels files. It then returns scores for the three metrics used in this track: P@5, MRR, and TBG. The metrics are reported for each context-profile pair and a mean score is also reported.
./eval.py runid.csv topics.csv desc-doc.qrels geo-nist.qrels geo-user.qrels
Submitted submissions must be in CSV format. If you have generated submissions in JSON format you can convert them with this script. The script takes in the name of your JSON submission file and saves a CSV with the same name but with '.csv' as the extension. This script requires that the 'ijson' Python packaged is installed.
As a warning, this script takes a few minutes to run.
./submission_json2csv.py submission.json
Only valid CSV submission files will be accepted. This is the script that is run on the submission server to check if a submission is valid. It takes in the name of your CSV submission file and saves a log file with the same but with '.errlog' appended to the end. If this file is empty and the script returns with a return code of 0 then the submission is valid, otherwise the log file will contain descriptions of the errors that need to be addressed.
./validate_submission.py submission.csv