MEDIQA-RADEVAL @ NAACL-ClinicalNLP 2027
Radiology Evaluation Metric across Modalities and Anatomies
This new shared task focuses on developing reliable methods for evaluating radiology report quality. Participants will use a new benchmark of expert-annotated report comparisons capturing clinically meaningful differences across imaging modalities and anatomical regions.
The dataset includes fine-grained annotations of relevant quality dimensions and error types for benchmarking automated evaluation metrics. By comparing metrics against expert assessments, the task aims to advance robust, clinically grounded evaluation of radiology report generation systems.
Step 1: Please fill out the MEDIQA-RADEVAL 2027 registration form at the following link: https://forms.cloud.microsoft/r/3JDJwg4wZJ
The registration must be submitted by the team lead or representative (please do not submit multiple registrations from the same team).
Step 2: After submitting the registration form, please review and accept the Terms, and join the Codabench competition using the following link: https://www.codabench.org/competitions/18515/ (You can find the Terms under the Terms page on Codabench).
Once both your registration form and your Codabench request have been approved, you will receive access to: The dataset & leaderboard on Codabench.
The ClinicalNLP 2027 workshop will recognize outstanding research contributions and top-performing shared task teams with several awards of $1,000 each:
MEDIQA-RADEVAL: Best-performing team.
Best Shared Task Paper: Innovative and insightful shared task research.
For more info: https://clinical-nlp.github.io/2027/awards.html
Registration opens on Codabench: Friday November 6, 2026
Development data release: Tuesday December 1st, 2026
Registration ends: Friday January 15, 2027
Test data release: Tuesday January 19, 2027
Run submission due: Friday January 22, 2027
Code submission due: Monday January 25, 2027
Release of the official results by the organizers: Friday January 29, 2027
Paper submission deadline: Wednesday February 24, 2027
Reviews due: Friday March 26, 2027
Notification of acceptance: Thursday April 1, 2027
Camera-ready paper due: Friday April 16, 2027
Pre-recorded video due: May 14, 2027
ClinicalNLP Workshop @ NAACL 2027: June 2027, San Francisco, California, United States
If you have any questions regarding your team's registration, please email us at mediqa.organizers@gmail.com
For more updates or inquiries, join the MEDIQA Google group https://groups.google.com/g/mediqa-nlp and email us at mediqa-nlp@googlegroups.com (mailing list)
Jean-Philippe Corbeil, Microsoft, Canada
Johannes Moll, Technical University of Munich, Germany
Ruochen Li, Technical University of Munich, Germany
Jingsong Liu, Technical University of Munich, Germany
Yuan Bi, Technical University of Munich, Germany
Juampablo E. Heras Rivera, University of Washington
Mehmet Kurt, University of Washington
Asma Ben Abacha, Microsoft, USA