data format

Run format and file description

* The run format should be the standard TREC format.

e.g.,

EN9991 Q0 EN12345678 1 0.9999 TRECRUN

EN9991 Q0 EN12345679 2 0.8888 TRECRUN

EN9991 Q0 EN12345670 3 0.7777 TRECRUN

EN9992 Q0 EN22345678 1 0.9999 TRECRUN

EN9992 Q0 EN22345679 2 0.8888 TRECRUN

EN9992 Q0 EN22345670 3 0.7777 TRECRUN

* Please use trec_eval to check your run format.

See) http://trec.nist.gov/trec_eval/

* In the standard 'trec_results', lines of results_file are:

030 Q0 ZF08-175-870 0 4238 prise1

qid iter docno rank sim run_id

For more details, comfirm 'trec_eval --help'.

In RecipeSearch, you may use the last column (the run_id part) to give brief explanation of the run.

* Max number of docs per topic to use in evaluation is 1000.

* Each file name should contain [1] Group ID, [2] Subtask Type, [3] Run Type, and [4] Run ID.

e.g.,

GROUP-EN1-BASE-01

GROUP-EN1-BASE-02

GROUP-JA1-TEST-01

GROUP-JA1-TEST-02

[1] Group ID: not exceed 5 alpha-numeric characters.

[2] Subtask Type: EN1, EN2, JA1, JA2.

[3] Run Type: BASE, ORCL, TEST.

[4] Run ID: two digit numbers starting from 01.

Subtask Type:

EN1 = English recipe ad hoc (subtask1)

EN2 = English recipe pairing (subtask2)

JA1 = Japanese recipe ad hoc (subtask1)

JA2 = Japanese recipe pairing (subtask2)

Run Type:

BASE = vanilla search. system input consists of only queries.

ORCL = vanilla search. system input consists of answer examples. (queries may be included.)

TEST = any input, any system. whatever you want to test. (queries and answer examples may be included.)

* Please fill out the following form, and send it to Michiko Yasukawa (michi [at] cs.gunma-u.ac.jp).

===== RUN submission =====

1. File name: ____________

2. MAP value: ____________

3. File description:

==========================

* Please send a run file as an attachment in your email message.

e.g.,

To: michi [at] cs.gunma-u.ac.jp

From: member [at] participants

Date: Fri, 01 Aug 2014 01:23:45 +0900

Subject: NTCIR-11 RecipeSearch run submission

===== RUN submission =====

1. File name: GROUP-EN1-TEST-01

2. MAP value: 0.001

3. File description:

BOOLEAN search using PostgreSQL9.4, no-stemming,

no-stopping. System input = queries, a hand-made dictionary.

==========================

add attachments to the message: [GROUP-EN1-TEST-01]

* You may submit multiple runs and multiple forms in your message.

Relevance assessment (by TOs and participants)

* Additional answers should be in the same format (CSV format) as the released example answers.

e.g.,

EN9991,EN98765432

EN9910,EN98765431

EN9100,EN98765430