MEDIQA-CORR@ ClinicalNLP 2024

MEDIQA-CORR @ NAACL-ClinicalNLP 2024

Medical Error Detection & Correction

Shared Task Paper: https://aclanthology.org/2024.clinicalnlp-1.57.pdf  

Dataset Paper (MEDEC): https://arxiv.org/pdf/2412.19260 

Motivation

Large language models (LLMs) show promise in being applied on unseen tasks with competitive ability.

However, by construction, such models have a key vulnerability; their ability is only as good as its underlying training data. Since LLMs rely on large corpora of textual data (often from the world wide web) for training, their data is almost impossible to manually curate at scale. If the data contains false information or only one perspective or type of information, the ability of LLMs to discern factual information may be hindered. Also, as a consequence to their own success, some online content may be entirely generated by LLMs that are prone to hallucinated information. In addition, in specialized domains, online information can be unreliable, harmful, and contain logical inconsistencies that may hinder the models' reasoning ability. However, most previous works on common sense detection have focused on the general domain [1-2].

In this task, we seek to address the problem of identifying and correcting (common sense) medical errors in clinical notes. From a human perspective, these errors require medical expertise and knowledge to be both identified and corrected. 

[1] SemEval-2020 Task 4: Commonsense Validation and Explanation. Cunxiang Wang, Shuailong Liang, Yili Jin, Yilong Wang, Xiaodan Zhu, Yue Zhang. 

[2] CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge. Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, Greg Durrett. 

Registration, Datasets & Evaluation 

• MEDIQA-CORR@Codabench: https://www.codabench.org/competitions/1900/ 

• MEDIQA-CORR@GitHub: https://github.com/abachaa/MEDIQA-CORR-2024

Registration:

1. Please complete the ClinicalNLP-MEDIQA 2024 registration form first: https://docs.google.com/forms/d/e/1FAIpQLScUnP2TJQX996BR-6dd6GvWAmkfE8VrX135I3VAYuecD1VR9Q/viewform?vc=0&c=0&w=1&flr=0

2. Download and fill out the Data Usage Agreement (DUA) and send the completed and signed DUA at mediqa.organizers@gmail.com by March 18, 2024, to have access to the UW datasets.

3. Accept the Terms and Conditions and join the Codabench project: https://www.codabench.org/competitions/1900/ 

Contact    

• If you have any questions regarding your team's registration, please email us at mediqa.organizers@gmail.com

• For more updates or inquiries, join the MEDIQA Google group https://groups.google.com/g/mediqa-nlp and email us at mediqa-nlp@googlegroups.com (mailing list)