Overview

Task 2 evaluates whether a system can determine the validity of a causal explanation, identify and classify hallucinations, correct the explanation, and provide a reliable confidence score.

The final score is calculated by adding the scores from five evaluation components.

1. Validity Classification — 10 Points

Each causal explanation is classified as valid or invalid.

Evaluation will report:

• precision;

• recall;

• F1; and

• balanced accuracy.

F1 will be used as the primary validity-classification score.

2. Error-Span Detection — 20 Points

Systems must identify the exact text span of each hallucination.

The primary metric is strict span-level precision, recall, and F1. A predicted span is counted as correct when its start and end boundaries match the gold span.

A relaxed overlap score may also be reported for error analysis, but it will not replace the strict official score.

For detailed rules, see the Span and Type Evaluation page.

3. Hallucination-Type Classification — 20 Points

Each detected error span must be assigned an official hallucination type.

A typed span is counted as correct when:

• the predicted span matches the gold span; and

• the hallucination type is correct.

The primary metric is typed span-level F1.

Macro-F1 by hallucination type will also be reported to show performance on less frequent error categories.

For detailed rules, see the Span and Type Evaluation page.

4. Correction Quality — 45 Points

The corrected causal explanation will be evaluated for:

• edit correctness;

• clinical factuality;

• avoidance of new hallucinations;

• minimality of unnecessary changes; and

• fluency and readability.

Correction evaluation will focus on the edited content rather than assigning a semantic-similarity score to the entire largely unchanged explanation.

For detailed rules, see the Correction Evaluation page.

5. Confidence Calibration — 5 Points

Participants must provide a confidence score from 0 to 1 for the validity judgment.

The primary calibration metric will be the Brier Score.

Expected Calibration Error and reliability diagrams may also be reported as supplementary analyses.

For clarity, confidence should represent:

The system’s estimated probability that its validity judgment is correct.

Task 2 Final Score

• Validity classification — 10 points

• Error-span detection — 20 points

• Hallucination-type classification — 20 points

• Correction quality — 45 points

• Confidence calibration — 5 points

Total: 100 points

Evidence-Use Categories

The same evaluation framework applies to all submitted runs.

Each run must indicate the evidence actually used:

• R — Report-only

• I — Image-only

• RI — Report-and-Image

The evidence-use category does not change the required output format or evaluation metrics. Results may also be grouped by evidence-use category for comparative analysis.