Task 1 Evaluation

Overview

Task 1 requires participants to generate a structured A1–A5 interpretation for each chest X-ray image.

Because the output consists of short structured answers rather than a free-text causal explanation, Task 1 will be evaluated using exact matching and set-based classification metrics.

Each A1–A5 component contributes 20 points to the final score.

A1 — Initial Impressions

A1 may contain multiple initial impressions.

Predictions will be evaluated using:

• precision;

• recall; and

• set-based F1.

Equivalent terms will be mapped to a canonical terminology before scoring.

A2 — Anatomical Location

Each location is a hierarchy of Level-1 (category), Level-2 (subcategory), and Level-3 (detail), chosen from the official A2 vocabulary. 

Scoring is hierarchical: a prediction earns partial credit for the levels it gets right, with deeper levels weighted more heavily, so a correct broad region still receives credit even when the finer level is wrong. Multiple locations are compared as a set. 

The evaluation will consider:

• anatomical region;

• lung zone or compartment, when applicable; and

• laterality.

A3 — Thoracic Level or Lung Zone

A3 is one or more thoracic-spine level ranges over T1–T12, each given as begin/end.

 Every range is expanded into the set of T-levels it covers, and scoring uses set overlap (level-set F1 / IoU), so a prediction that is close — for example, off by one level — still receives partial credit. 

Multiple ranges are combined into a single level set before scoring. 

A4 — Final Impressions

A4 may contain one or more final impressions, and an empty A4 (the official normal label) indicates a normal case.

Label agreement is scored with micro-F1 (primary) and macro-F1.

In addition, the normal-vs-abnormal decision is scored with balanced accuracy, so that missing a truly abnormal case is penalized appropriately on imbalanced data.

The A4 score combines the label score and the normal/abnormal score.

A5 — Confirmation Checklist

A5 links each A4 finding to its confirmation-checklist item numbers. 

It is scored as (finding, item) pairs using pair-based F1, so both the checklist selection and its link to the correct finding must be right. 

A5 is scored against the reference findings, so a mistake in A4 does not doubly penalize A5. 

Terminology Normalization

Before evaluation, predictions may be normalized for:

• capitalization;

• leading and trailing spaces;

• punctuation;

• approved abbreviations;

• singular and plural forms; and

• officially recognized synonyms.

The official vocabulary and normalization mapping will be released with the evaluation script.

Task 1 Final Score

• A1 — 20 points

• A2 — 20 points

• A3 — 15 points

• A4 — 25 points

• A5 — 20 points

Total: 100 points