MEDIQA-EVAL @ ClinicalNLP 2026
Evaluating Metrics for Multimodal Question Answering
Digital messaging platforms have become an integral part of remote patient care, improving access while reducing costs. However, they also increase the workload for healthcare providers. Patient questions in visually driven specialties such as dermatology [1] and wound care [2] often include clinical images. Automatically generating draft responses to these multimodal queries offers a promising way to ease this burden.
Although Vision language models (VLMs) can produce fluent, human-like text, reliably evaluating the accuracy and completeness of their outputs remains challenging, particularly in multimodal settings where images must be interpreted alongside textual information and where multiple valid gold responses may exist due to differing expert opinions [3].
We extend the MEDIQA-M3G 2024 [4], MEDIQA-WV 2025 [5], MEDIQA-MAGIC 2024 [6] & 2025 [7] Shared Tasks on dermatology and wound-care visual question answering by introducing a new focus on evaluating open-ended system responses. In this multimodal evaluation task, participants assign quality scores to model-generated answers for patient questions that are paired with one or multiple clinical images.
The dataset includes both English and Chinese questions and answers, along with structured metadata such as anatomical location and wound type. Each system answer is independently rated by clinical experts along three dimensions: overall quality, factual accuracy, and completeness.
References:
DermaVQA: A Multilingual Visual Question Answering Dataset for Dermatology. Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Asma Ben Abacha, Meliha Yetisgen, Fei Xia. MICCAI (5) 2024: 209-219
WoundcareVQA: A multilingual visual question answering benchmark dataset for wound care. Wen-wai Yim, Asma Ben Abacha, Robert Doerning, Chia-Yu Chen, Jiaying Xu, Anita Subbarao, Zixuan Yu, Fei Xia, M. Kennedy Hall, Meliha Yetisgen. J. Biomed. Informatics 170: 104888 (2025)
MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering. Wen-wai Yim, Asma Ben Abacha, Zixuan Yu, Robert Doerning, Fei Xia, Meliha Yetisgen. CoRR abs/2509.12405
Overview of the MEDIQA-M3G 2024 Shared Task on Multilingual Multimodal Medical Answer Generation. Wen-wai Yim, Asma Ben Abacha, Yujuan Fu, Zhaoyi Sun, Fei Xia, Meliha Yetisgen, Martin Krallinger. ClinicalNLP@NAACL 2024: 581-589
Overview of the MEDIQA-MAGIC Task at ImageCLEF 2024: Multimodal And Generative TelemedICine in Dermatology. Wen-wai Yim, Asma Ben Abacha, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia. CLEF (Working Notes) 2024: 1456-1462
Step 1: Please fill out the MEDIQA-EVAL 2026 registration form at the following link: https://forms.gle/FHiTycN2GJ9TF5Nv5
If you are unable to access the registration form, please email us at mediqa.organizers@gmail.com with the following information: {first name, last name, team name, affiliation, country/countries of affiliation, codabench username (for run submission), github username (for code submission)}
The registration must be submitted by the team lead or representative (please do not submit multiple registrations from the same team).
Step 2: After submitting the registration form, please review and accept the Terms, and join the Codabench competition using the following link: https://www.codabench.org/competitions/12115/ (You can find the Terms under the Terms page on Codabench).
Once both your registration form and your Codabench request have been approved, you will receive access to the dataset & leaderboard on Codabench.
All deadlines are 11:59PM UTC-12:00 (anywhere on Earth)
First CFP & Registration opens: Mon 8 Dec 2025
Development data release: Mon 15 Dec 2025
Registration ends: Fri 30 Jan 2026
Test data release: Mon 02 Feb 2026
Run (& code) submission due: Thu 05 Feb 2026
Release of the results by the organizers: Fri 06 Feb 2026
Paper submission due: Fri 13 Feb 2026
Notification of acceptance: Wed 11 Mar 2026
Final version due: Mon 30 Mar 2026
ClinicalNLP Workshop: Saturday 16 May 2026, Palma de Mallorca, Spain
If you have any questions regarding your team's registration, please email us at mediqa.organizers@gmail.com
For more updates or inquiries, join the MEDIQA Google group https://groups.google.com/g/mediqa-nlp and email us at mediqa-nlp@googlegroups.com (mailing list)