MEDIQA-MULTIREFEVAL @ NAACL-ClinicalNLP 2027
Evaluating Metrics for Multimodal Question Answering
The second edition of MEDIQA-EVAL [1] focuses on evaluating open-ended responses in dermatology visual question answering, where generated answers can assist clinicians by providing draft responses to patient questions accompanied by clinical images. Participants will develop metrics for assessing answers against expert judgments.
The task will feature a larger multilingual English and Chinese dataset, with four additional gold-reference answers per language and expert ratings for generated and reference answers. These annotations will support evaluation when multiple clinically valid responses exist. Building on the first edition, the task aims to advance clinically grounded metrics capturing factual accuracy, completeness, and overall response quality across languages and dermatology cases.
References
[1] Overview of the MEDIQA-EVAL 2026 Shared Task on Evaluation Metrics in Medical Multimodal Question Answering. Asma Ben Abacha, Wen-Wai Yim. Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026
Step 1: Please fill out the MEDIQA-MULTIREFEVAL 2027 registration form at the following link: https://forms.cloud.microsoft/r/kFMP7m30hn
The registration must be submitted by the team lead or representative (please do not submit multiple registrations from the same team).
Step 2: After submitting the registration form, please review and accept the Terms, and join the Codabench competition using the following link: https://www.codabench.org/competitions/18491 (You can find the Terms under the Terms page on Codabench).
Once both your registration form and your Codabench request have been approved, you will receive access to: The dataset & leaderboard on Codabench.
The ClinicalNLP 2027 workshop will recognize outstanding research contributions and top-performing shared task teams with several awards of $1,000 each:
MEDIQA-MULTIREFEVAL (English): Best-performing team.
MEDIQA-MULTIREFEVAL (Chinese): Best-performing team.
Best Shared Task Paper: Innovative and insightful shared task research.
For more info: https://clinical-nlp.github.io/2027/awards.html
Registration opens on Codabench: Friday November 6, 2026
Development data release: Tuesday December 1st, 2026
Registration ends: Friday January 15, 2027
Test data release: Tuesday January 19, 2027
Run submission due: Friday January 22, 2027
Code submission due: Monday January 25, 2027
Release of the official results by the organizers: Friday January 29, 2027
Paper submission deadline: Wednesday February 24, 2027
Reviews due: Friday March 26, 2027
Notification of acceptance: Thursday April 1, 2027
Camera-ready paper due: Friday April 16, 2027
Pre-recorded video due: May 14, 2027
ClinicalNLP Workshop @ NAACL 2027: June 2027, San Francisco, California, United States
If you have any questions regarding your team's registration, please email us at mediqa.organizers@gmail.com
For more updates or inquiries, join the MEDIQA Google group https://groups.google.com/g/mediqa-nlp and email us at mediqa-nlp@googlegroups.com (mailing list)