Background

We propose a shared task on in-context compositionality prediction: given a target multiword expressions (a noun compound such as glass ceiling or a particle verb such as take off) and a sentence in which that expression is used, systems are expected to output numerical scores reflecting human ordinal compositionality ratings for the target expression in that context.

Our task addresses the compositionality prediction problem from several novel angles. We incorporate two types of multiword expressions (noun compounds and particle verbs) in two languages (English and German). For each of the 1,539 target expressions, we draw on two types of corpora (present-day and historical data) to provide ≈10 example sentences, labeling each with human ordinal compositionality ratings. 

These properties ensure a challenging task: because of the token-level, ranking-task framing; the diversity of MWE types and languages; and the entirely novel use of historical data, which requires robustness to the evolution of target expressions and their constituents, best observed in sparse (therefore additionally challenging) resources.

Important dates

September 5, 2026: Trial data release
September 8, 2026: Training data release
January 10, 2027: Evaluation start
January 31, 2027: Evaluation end

How to participate

The task is run on CodaBench:
https://www.codabench.org/competitions/17971/

Please refer to the CodaBench competition page for details on the data format, submission instructions, and further practical details.

Organizers

Filip Miletić, Chris Jenkins, Sabine Schulte im Walde
University of Stuttgart

Contact: semeval-compartment (at) ims (dot) uni-stuttgart (dot) de