Presupposed Taxonomies: Evaluating Neural Network Semantics (PreTENS)

NOTICE: The evaluation phase has ended! Test labels/scores are now available.
Winners for the tasks will be announced soon!
!!! Paper submission deadline: 28th Februrary using OpenReview (Submission Site) !!!

A growing body of literature on the cognitive plausibility of computational linguistics has tried to probe the ability of modern language models to recognize linguistic structures that are deviant at the syntactic and/or semantic level. An active subfield of this research concerns the ability of these models to recognize lexical relationships between words and to demonstrating a capacity for generalization.

The PreTENS shared task hosted at SemEval 2022 aims at focusing on semantic competence with specific attention on the evaluation of language models with respect to the recognition of appropriate taxonomic relations between two nominal arguments (i.e. cases where one is a supercategory of the other, or in extensional terms, one denotes a superset of the other). For instance animal is a supercategory of dog.

The complicating factor is that there are many 2 argument constructions in human language which impose a constrain on these taxonomic relations and on their order. This constraint can be seen as a presupposition imposed by the construction. For instance, I have a GUN, not a WEAPON violates the prosupposition that the two uppercased arguments should NOT be in a taxonomic relation (since guns are weapons). Models attempting this shared task thus face a double challenge: they have to recognize whether a taxonomic relation holds between the two nominals in the sentence, and they have to recognize whether the construction which embeds the nominals is one which accepts the relation (in a given order), or one which disallows it.

For this purpose, we introduce a novel dataset, containing a broad range of presuppositional constructions labeled with acceptability level. Thanks to the fact that the data are programmatically generated from a human-verified template, we propose the dataset in multiple languages, i.e. English, Italian and French, with minimal variation.

Beyond Natural Language Inference practitioners, we believe the task should attract researchers interested in the potential of NNs as cognitive/linguistic models, or more generally looking for tools to probe into the black box of many current language models.

Please fill out this form to participate.

See the Task description page for more information.

Evaluation Platforms:
Subtask1: https://codalab.lisn.upsaclay.fr/competitions/1292
Subtask2: https://codalab.lisn.upsaclay.fr/competitions/1290

Test data in: https://github.com/shammur/SemEval2022Task3

Important dates

• Trial data available: 31st July 2021

• Training data available: 3rd September 2021

• Evaluation Period [Done]: 18th January 2022 - 22nd January 2022 (AOE)

• Abstract Submission: 8th February 2022**

• Final Paper Submission Due: 28th February 2022

• Notification to Authors: 31 March 2022

Paper Submission Details

• Abstract Submission: 8th February 2022

For submitting the abstract send an email to the semeval2022-task3-organizers@googlegroups.com with the detailed team name, along with author names.
SemEval paper titles follow a fixed template, Please se below for details.

• Paper Title format (for SemEval)

System description papers for one task (any number of subtasks) are titled "Team Name at SemEval-YYYY Task N: Descriptive Title"

• Note "at", not "@"

• Note that "SemEval" and the year are separated by a hyphen and no spaces, e.g. "SemEval-2020"

• Note that the task number is followed by a colon, not a dash

• Note that the colon has a space after it but not before it

Authors are free to choose the Descriptive Title as they would a normal paper title; it may mention a particular question addressed, method used, or finding discussed in the paper

System description papers for multiple tasks are titled "Team Name at SemEval-YYYY Tasks N1 and N2: Descriptive Title" (2 tasks) or "Team Name at SemEval-YYYY Tasks N1, N2, and N3: Descriptive Title" (3 or more tasks)

• Paper Submission ( Instruction to be updated later)
  
  Paper style format: NAACL 2022
  Length of the paper:
  System paper submissions for a single task can be up to 5 pages. Some teams participating in multiple related (sub)tasks choose to publish a single system paper. In this case, the limit is 8 pages.
  Acknowledgments, references, and appendices do NOT count toward page limits. (for more details: https://semeval.github.io/paper-requirements.html)