Task Definition

TL;DR

The input is medical text describing the clinical history of a patient with Dyspnea (a respiratory medical condition). The task is to assign a value to a list of items (e.g., temperature, blood pressure, ...) based on the patient's medical history. The list of items to populate is called the Dyspnea Case Report Form (CRF).

Participants are given: a) few gold-standard populated Dyspnea CRFs, b) some semi-automatically populated CRFs for several medical conditions, c) thousands of clinical notes of patients with Dyspnea.

The task is released in two languages: English and Italian. Participants are free to submit on one language or both languages.

Task

Given a clinical note, the task consists of populating the Dyspnea Case Report Form (CRF). The CRF consists of a predefined list of 134 items. Each item must be populated based on the information contained in the clinical note. The same list of items applies to all clinical notes. The Dyspnea CRF can be found here. Each item has a list of valid options to use to populate it.

The task is proposed in two languages, English and Italian. Parallel data is released for training, development, and testing in both languages (more information available here).

There are two language-specific leaderboards, one per language. We encourage participants to test their methods and systems on both languages, by running them on both Italian and English data, and submit their results to both leaderboards.

Data sparcity

Annotations are very sparse, as often times the CRF item can not be populated based on the information contained in the clinical note. In such cases, the item is filled with an "unknown" value. This happens 95% of the times. This represents one of the main challenges of the task.

‼️Example‼️
------------------------------------

Here we report an example. For a description of the datasets, please refer to the dataset page.

CLINICAL NOTE:

CRF (items set) with gold-standard annotation

📊 EVALUATION 📊

The evaluation metric is macro F1. The evaluation script, including the metric calculation, is reported at this link.

Page updated

Report abuse