Task 2 Evaluation

Overview

Task 2 evaluates whether a system can determine the validity of a causal explanation, identify and classify hallucinations, correct the explanation, and provide a reliable confidence score.

The final score is calculated by adding the scores from five evaluation components.

1. Validity Classification — 15 Points

Each causal explanation is classified as valid or invalid.

Evaluation will report:

• precision;

• recall;

• F1; and

• balanced accuracy.

F1 will be used as the primary validity-classification score.

2. Error-Span Detection — 20 Points

The primary metric is a partial-match span F1. 

A span that exactly matches a reference boundary is considered fully correct, while a span that overlaps a reference span receives partial credit.

A strict exact-boundary F1 and a character-overlap F1 are also reported for analysis.

For detailed rules, see the Span and Type Evaluation page.

3. Hallucination-Type Classification — 20 Points

Each detected error span must be assigned an official hallucination type.

A typed span is counted as correct when:

• the predicted span overlaps the gold span;  and

• the hallucination type is correct.

The primary metric is a typed partial-match F1.

Macro-F1 by hallucination type will also be reported to show performance on less frequent error categories.

For detailed rules, see the Span and Type Evaluation page.

4. Correction Quality — 45 Points

The corrected causal explanation will be evaluated for:

• edit correctness;

• clinical factuality;

• avoidance of new hallucinations;

• minimality of unnecessary changes; and

• fluency and readability.

Correction evaluation will focus on the edited content rather than assigning a semantic-similarity score to the entire largely unchanged explanation.

For detailed rules, see the Correction Evaluation page.

Task 2 Final Score

• Validity classification — 15 points

• Error-span detection — 20 points

• Hallucination-type classification — 20 points

• Correction quality — 45 points

Total: 100 points

Evidence-Use Categories

The same evaluation framework applies to all submitted runs.

Each run must indicate the evidence actually used:

• R — Report-only

• I — Image-only

• RI — Report-and-Image

The evidence-use category does not change the required output format or evaluation metrics. Results may also be grouped by evidence-use category for comparative analysis.