data format
Run format and file description
* The run format should be the standard TREC format.
e.g.,
EN9991 Q0 EN12345678 1 0.9999 TRECRUN
EN9991 Q0 EN12345679 2 0.8888 TRECRUN
EN9991 Q0 EN12345670 3 0.7777 TRECRUN
EN9992 Q0 EN22345678 1 0.9999 TRECRUN
EN9992 Q0 EN22345679 2 0.8888 TRECRUN
EN9992 Q0 EN22345670 3 0.7777 TRECRUN
* Please use trec_eval to check your run format.
See) http://trec.nist.gov/trec_eval/
* In the standard 'trec_results', lines of results_file are:
030 Q0 ZF08-175-870 0 4238 prise1
qid iter docno rank sim run_id
For more details, comfirm 'trec_eval --help'.
In RecipeSearch, you may use the last column (the run_id part) to give brief explanation of the run.
* Max number of docs per topic to use in evaluation is 1000.
* Each file name should contain [1] Group ID, [2] Subtask Type, [3] Run Type, and [4] Run ID.
e.g.,
GROUP-EN1-BASE-01
GROUP-EN1-BASE-02
GROUP-JA1-TEST-01
GROUP-JA1-TEST-02
[1] Group ID: not exceed 5 alpha-numeric characters.
[2] Subtask Type: EN1, EN2, JA1, JA2.
[3] Run Type: BASE, ORCL, TEST.
[4] Run ID: two digit numbers starting from 01.
Subtask Type:
EN1 = English recipe ad hoc (subtask1)
EN2 = English recipe pairing (subtask2)
JA1 = Japanese recipe ad hoc (subtask1)
JA2 = Japanese recipe pairing (subtask2)
Run Type:
BASE = vanilla search. system input consists of only queries.
ORCL = vanilla search. system input consists of answer examples. (queries may be included.)
TEST = any input, any system. whatever you want to test. (queries and answer examples may be included.)
* Please fill out the following form, and send it to Michiko Yasukawa (michi [at] cs.gunma-u.ac.jp).
===== RUN submission =====
1. File name: ____________
2. MAP value: ____________
3. File description:
==========================
* Please send a run file as an attachment in your email message.
e.g.,
To: michi [at] cs.gunma-u.ac.jp
From: member [at] participants
Date: Fri, 01 Aug 2014 01:23:45 +0900
Subject: NTCIR-11 RecipeSearch run submission
===== RUN submission =====
1. File name: GROUP-EN1-TEST-01
2. MAP value: 0.001
3. File description:
BOOLEAN search using PostgreSQL9.4, no-stemming,
no-stopping. System input = queries, a hand-made dictionary.
==========================
add attachments to the message: [GROUP-EN1-TEST-01]
* You may submit multiple runs and multiple forms in your message.
Relevance assessment (by TOs and participants)
* Additional answers should be in the same format (CSV format) as the released example answers.
e.g.,
EN9991,EN98765432
EN9910,EN98765431
EN9100,EN98765430