ASSIN 2

Contextualization

ASSIN 2 is the second edition of the Avaliação de Similaridade Semântica e Inferência Textual (Evaluating Semantic Similarity and Textual Entailment), and was a workshop collocated with STIL 2019. It follows the first edition of ASSIN, proposing a new shared task with new data.

The workshop evaluated systems that assess two types of relations between two sentences: Semantic Textual Similarity and Textual Entailment.

Semantic Textual Similarity consists of quantifying the level of semantic equivalence between sentences, while Textual Entailment Recognition consists of classifying whether the first sentence entails the second.

Data

The corpus used in ASSIN 2 is composed of rather simple sentences. Following the procedures of SemEval 2014 Task 1, we tried to remove from the corpus named entities and indirect speech, and tried to have all verbs in the present tense. The annotation instructions given to annotators are available (in Portuguese).

The training and validation data are composed, respectively, of 6,500 and 500 sentence pairs in Brazilian Portuguese, annotated for entailment and semantic similarity. Semantic similarity values range from 1 to 5, and text entailment classes are either entailment or none. The test data are composed of approximately 3,000 sentence pairs with the same annotation. All data were manually annotated.

• Training data (no validation set)
• Validation data
• Test data (blind)
• Test data (full)

Evaluation

The evaluation of submissions to ASSIN 2 was with the same metrics as the first ASSIN, with the F1 of precision and recall as the main metric for text entailment and Pearson correlation for semantic similarity. The evaluation scripts are the same as in the last edition.

Discussion Forum

https://groups.google.com/forum/#!forum/assin2

Important Dates

• Call for pariticipation --- May/June 2019
• Training data avaliable --- June 6 June 16
• Test data available --- September 6 September 16
• Systems outputs due --- September 16 September 26
• Workshop --- October 15-18
• First paper submission --- TBA
• Camera ready papers --- TBA

Organization

Livy Real (B2W Digital/Grupo de Linguística Computacional - USP)

Hugo Gonçalo Oliveira (CISUC, Universidade de Coimbra)

Erick Fonseca (Instituto de Telecomunicações, Lisboa)

Publications

• Real, Livy; Fonseca, Erick; Gonçalo Oliveira, Hugo. The ASSIN 2 shared task: a quick overview (STIL 2019)

Acknowledgements

We would like to express the deepest appreciation to Alissa Canesim (Universidade Estadual de Ponta Grossa), Amanda Oliveira (Stilingue), Ana Cláudia Zandavalle (Stilingue), Beatriz Mastrantonio (Stilingue - Universidade Federal de Ouro Preto), Carolina Gadelha (Stilingue), Denis Owa (Pontifícia Universidade Católica de São Paulo), Evandro Fonseca (Stilingue), Marcos Zandonai (Universidade do Vale do Rio dos Sinos), Renata Ramisch (Núcleo Interinstitucional de Linguística Computacional - Universidade Federal de São Carlos), Talia Machado (Universidade Estadual de Ponta Grossa) for taking part of this task with the single purpose of producing open resources for the community interested on the computational processing Portuguese. We also thank the Group of Computational Linguistics from University of São Paulo for making available SICK-BR, which served as the base for the ASSIN corpus.

Results

The table below includes a tab "hard pairs", where results in supposedly harder pairs are displayed. These pairs were obtained as follow. A baseline classifier was trained for textual inference recognition having access only to the second sentence of each pair. The pairs incorrectly classified by it were tagged as hard, and a list with their ids can be found here. Note that it is possible to reach a different set of such pairs initializing the classifier with a different random seed.

Picture Credits: Márcio Filho - MTUR