Correction Evaluation

Overview

Correction evaluation determines whether a system successfully repairs the identified errors while preserving correct content.

Whole-text semantic similarity alone is insufficient because most of the causal explanation may remain unchanged. A system could receive a high similarity score even when the actual correction is wrong.

Correction evaluation therefore focuses on the edits made by the system and on the clinical correctness of the final explanation.

1. Edit Correctness — 15 Points

System edits will be compared with one or more gold corrections.

Evaluation may include:

• exact edit match;

• token-level F1 within the corrected span; and

• edit-level semantic similarity.

BERTScore or a biomedical sentence-similarity metric may be applied to the corrected span or sentence rather than to the entire explanation.

When multiple corrections are clinically acceptable, the system output may be compared against multiple reference corrections.

2. Clinical Factuality — 20 Points

The corrected explanation must be clinically consistent with the available evidence.

Evaluation will consider:

• correctness of radiological findings;

• anatomical and laterality accuracy;

• diagnostic consistency;

• correctness of causal relationships;

• consistency with the report, image, or both; and

• completeness of clinically important evidence.

Radiology-specific entity and relation metrics may be used as supporting measures.

An evidence-grounded rubric or radiology-specific evaluator may also be used to identify clinically significant errors that semantic similarity does not capture.

3. No-New-Hallucination and Minimality — 5 Points

A correction must not introduce new unsupported findings, diagnoses, or causal relationships.

Unnecessary changes to correct portions of the original explanation may receive a penalty.

For a valid input, the explanation should remain unchanged except for permitted formatting normalization.

4. Fluency and Readability — 5 Points

The corrected explanation should be:

• grammatically well formed;

• coherent;

• understandable; and

• appropriate for a clinical explanation.

Fluency will be evaluated separately from semantic similarity and clinical factuality.

Correction Score

• Edit correctness — 15 points

• Clinical factuality — 20 points

• No-new-hallucination and minimality — 5 points

• Fluency and readability — 5 points

Total: 45 points

Expert Review

Selected top-performing runs may undergo expert review by radiologists.

Expert review may be used to examine:

• clinically important errors not captured by automatic metrics;

• alternative but acceptable corrections;

• unsupported new claims; and

• the safety and readability of the final explanation.

Expert review should be used as validation or tie-breaking rather than as the only evaluation method.