Span and Type Evaluation

Overview

Error-span detection and hallucination-type classification are evaluated using methods adapted from named entity recognition.

Each gold error is represented by:

• a start position;

• an end position; and

• a hallucination-type label.

A predicted error span is counted as correct only when its start and end boundaries exactly match a gold span.

Strict span precision, recall, and F1 are calculated as follows:

• Precision: the proportion of predicted spans that exactly match a gold span

• Recall: the proportion of gold spans that are exactly detected

• F1: the harmonic mean of precision and recall

Strict span F1 is the primary metric for error-span detection.

A relaxed score may be calculated when a predicted span overlaps a gold span but does not exactly match its boundaries.

Relaxed overlap evaluation is intended for error analysis. It will not replace strict span F1 in the official score.

The official script will define how overlapping and nested predictions are matched.

A typed span is counted as correct only when:

• its boundaries match the gold span; and

• its hallucination-type label matches the gold label.

Typed span precision, recall, and F1 will be used as the primary joint measure of span and type performance.

Macro-F1 will be reported across the official hallucination types:

• ADD-PATH

• LOC-LAT

• SEV-CHG

• NEG-FLIP

• CAUSE-WRONG

• DDX-WRONG

• EVID-OMIT

• CONTRA

A confusion matrix may also be provided to show which hallucination types are commonly confused.

When a causal explanation contains multiple errors, each error must be submitted as a separate span-and-type pair.

Duplicate predictions for the same gold span will not receive additional credit.

Predicted spans that do not correspond to a gold error will be counted as false positives.

For a valid causal explanation, the correct error-span output is “none.”

Any error span predicted for a valid case will be counted as a false positive.

Page updated

Google Sites

Report abuse