PIR-CLEF Medical Search Challenge 2019
Medical search is one of the most common interests of users of search engines. For this year’s pilot task, we challenge participants to work on the task of generating personalized retrieval techniques for the queries posed by patients on viewing their discharge summaries (this topic set and associated discharge summaries are described below), where the discharge summaries are used in this personalization process. Optionally additional resources (ontologies) can also be used in retrieval technique generation. Participants can submit any type of run they want to the challenge, so long as it is somehow personalized. This pilot challenge uses test collections originating from CLEF eHealth 2013 and 2014 information retrieval (IR) challenges.
**Test data now available at https://github.com/lorraine-goeuriot/PIR-health-CLEF - see details below**
Dataset Details:
Document collection:
The data set for this task consists of a set of medical-related documents, provided by the Khresmoi project. This collection contains documents covering a broad set of medical topics, and does not contain any patient information. The documents in the collection come from several online sources, including the Health On the Net organization certified websites, as well as well-known medical sites and databases (e.g. Genetics Home Reference, ClinicalTrial.gov, Diagnosia).
Document collection is available here: http://catalog.elra.info/en-us/repository/browse/ELRA-E0042/
Topics (CLEF eHealth 2013-2014 topics):
The topic set consists of 50 topics which were previously distributed with CLEF eHealth 2013 IR challenge and 50 topics which were distributed with CLEF eHealth 2014 IR challenge.
The 50 2013 topics were manually generated by medical professionals from highlighted disorders identified in annotated medical discharge summaries. In 2014, the 50 topics were manually generated by medical professionals from the main disorder diagnosed in the discharge summary. A mapping between queries and task 1 matching discharge summary is provided, the participants can get access to the discharge summary (from the MIMIC II database) – details provided below.
Topics consist of:
● Title: text of the query,
● Description: longer description of what the query means,
● Narrative: expected content of the relevant documents,
● Profile: main information on the patient (age, gender, condition)
● Discharge_summary: ID of the matching discharge summary
Training & Test Topics are available here: https://github.com/lorraine-goeuriot/PIR-health-CLEF
Personalization Material - Discharge summaries:
For this year’s 2019 challenge, participants are challenged with using the discharge summaries for personalization.
The discharge summaries consist of deidentified clinical free-text notes from the MIMIC II database, version 2.5 (mimic.physionet.org). Notes were authored in the ICU setting and note types include discharge summaries, ECG reports, echo reports, and radiology reports (for more information about the MIMIC II database, we refer the reader to the MIMIC User Guide).
PIR-CLEF organizers are not providing direct access to the discharge summaries, participants must follow MIMIC II guidelines to access it: https://mimic.physionet.org/gettingstarted/access/
Relevance judgments:
Relevance judgments (Qrel files) created by the CLEF eHealth challenge, which will also be used in this year's challenge, are available here. Details on how the qrel files were generated are available in the CLEF eHealth IR 2013 and CLEF eHealth IR 2014 overview papers.
2019 Challenge:
2019 Training/test topics:
● 40 training queries: available here https://github.com/lorraine-goeuriot/PIR-health-CLEF . This consists of 20 queries from the CLEF eHealth 2013 topic set + 20 from the CLEF eHealth 2014 topic set.
● 60 test queries: available here https://github.com/lorraine-goeuriot/PIR-health-CLEF . This consists of 30 from the CLEF eHealth 2013 topic set + 30 from the CLEF eHealth 2014 topic set. Note - participants should stop system development before running the test queries, and no further system development is allowed following release of test queries.
2019 Subtasks:
- Run 1 (mandatory) is a baseline: only title in the query can be used, and no external resource (including discharge summary, corpora, ontology, etc) can be used
- Runs 2-4 (optional): any experiment using the topic title and additional external resources (corpora, ontologies, etc. - see end of page for link to sample resources) but NOT the discharge summaries
- Runs 5-7 (optional): any experiment using the topic titles and summary information available in the topic but NOT the discharge summaries (additional external resources can also be used in this experiment)
- Runs 8-10 (optional): any experiment using the discharge summaries
One of the runs from 2-4 and one from 5-7 and one from 8-10 must use the IR technique used in Run 1. The idea being to allow analysis of the impact of discharge summaries/other techniques on the performance of the baseline Run 1.
The optional runs must be ranked in order of priority (for Runs 2-4, 2 is the highest priority; for Runs 5-7, 5 is the highest priority; for Runs 8-10, 8 is the highest priority).
Submitted runs should use the following naming convention: <TeamName>_Run<RunNumber>.<FileFormat>
For example: MU_Run1.dat , MU_Run5.dat
Evaluation Metrics:
Evaluation will focus on P@5, P@10, NDCG@5, NDCG@10.
Evaluation metrics can be computed with the trec_eval evaluation tool, which is available from http://trec.nist.gov/trec_eval/.
Working notes:
Participating groups in this task must submit a report (working notes) describing their task experiments.
Details on preparing working notes & link to the working notes submission system available on the CLEF 2019 website: http://clef2019.clef-initiative.eu/index.php
Runs Submission Instructions:
Participants should stop developing their systems when the test queries are released, and run the test queries on their developed systems and calculate the evaluation metrics using the available test queries relevance assessments (qrels files).
By the submission deadline of 22nd May the following should be submitted:
(1) run files
(2) calculated evaluation metrics for each run (i.e. P@10, etc.)
(3) a short paragraph summary of the IR techniques that were used for each run. they used (for our overview paper).
Runs, results and run summary information should be submitted as a single zip file. To submit this information please email liadh.kelly <at> mu.ie and lorraine.goeuriot <at> imag.fr for submission link.
Important Dates:
- Training Data Released: 8 April 2019
- Test Data Released [no further system development allowed]: 14 May 2019
13 May 2019 - Run, results, and run summary Submission Deadline: 22 May 2019
20 May 2019 - Participants’ working notes papers submitted [CEUR-WS]: 31 May 2019
24 May 2019 - Notification of Acceptance Participant Papers [CEUR-WS]: 14 June 2019
- Camera Ready Copy of Participant Papers [CEUR-WS] due: 29 June 2019
- Additional Resources:
(1) the CLEF eHealth 2013 and 2014 IR overview papers, these include details on the best submissions to these challenges (and references to the associated papers):
(2) links to additional medical resources that might prove useful in IR technique development:
https://www.nlm.nih.gov/research/umls/
(3) See https://sites.google.com/site/shareclefehealth/ and https://sites.google.com/site/clefehealth2014/ for full details on the CLEF eHealth 2013 and 2014 IR challenges.
Challenge Leaders:
- Lorraine Goeuriot, Université Grenoble Alpes, Grenoble, France
- Liadh Kelly, Maynooth University, Ireland