Span and Type Evaluation

Overview

Error-span detection and hallucination-type classification are evaluated using methods adapted from named entity recognition.

Each gold error is represented by:

• a start position;

• an end position; and

• a hallucination-type label.

Strict Span Matching

A predicted error span is counted as correct only when its start and end boundaries exactly match a gold span.

Strict span precision, recall, and F1 are calculated as follows:

• Precision: the proportion of predicted spans that exactly match a gold span

• Recall: the proportion of gold spans that are exactly detected

• F1: the harmonic mean of precision and recall

Strict span F1 is reported as a reference (strict) score, not the primary metric.

Relaxed Span Matching

AThe primary metric is a partial-match span F1: 

a span whose boundaries exactly match a gold span is fully correct, and a span that overlaps a gold span receives half credit.

A strict exact-boundary F1 and a character-overlap F1 are also reported for analysis.

The official script will define how overlapping and nested predictions are matched.

Typed Span Matching

A typed span is counted as correct only when:

• it overlaps the gold span (partial match); and

• its hallucination-type label matches the gold label.

Typed partial-match precision, recall, and F1 are the primary joint measure of span and type performance.

Type-Level Analysis

Macro-F1 will be reported across the official hallucination types:

• ADD-PATH

• ADD-DEVICE

• NEG-FLIP

• LAT-FLIP

• CONTRA

A confusion matrix may also be provided to show which hallucination types are commonly confused.

Multiple Errors

When a causal explanation contains multiple errors, each error must be submitted as a separate span-and-type pair.

Duplicate predictions for the same gold span will not receive additional credit.

Predicted spans that do not correspond to a gold error will be counted as false positives.

Valid Cases

For a valid causal explanation, the correct error-span output is “none.”

Any error span predicted for a valid case will be counted as a false positive.