About the Shared Task

An obstacle of scientific document understanding (SDU) is the extensive use of acronyms which are shortened forms of long technical phrases. In order to correctly process a document, an SDU system should be able to identify acronyms and their correct meanings. As acronyms might be defined either locally in the same document or globally in an external dictionary with multiple meanings, a successful document understanding model should both capture local definitions and disambiguate acronyms which are not defined in documents. To push forward the research on acronym understanding, we propose two shared tasks at SDU@AAAI-21.

Task 1 - Acronym Identification

This task aims to identify acronyms (i.e., short-forms) and their meanings (i.e., long-forms) from the documents. For instance:

Input: Existing methods for learning with noisy labels (LNL) primarily take a loss correction approach.

Output: Existing methods for learning with noisy labels (LNL) primarily take a loss correction approach.

In this example, the acronym is shown in bold font and the long-form is shown with underline. This task is modeled as a sentence level sequence labeling problem. Participants are provided with manually labeled training and development datasets consisting of 17,506 sentences extracted from English scientific papers published at arXiv. The submitted systems will be evaluated based on their precision, recall and F1 scores on the hidden test set computed for correct predictions of short-form (i.e., acronym) and long-form (i.e., phrase) boundaries in the sentences. The corpus and evaluation scripts can be downloaded from https://github.com/amirveyseh/AAAI-21-SDU-shared-task-1-AI

Task 2 - Acronym Disambiguation

This task aims to find the correct meaning of an ambiguous acronym in a given sentence. The input to the system is a sentence with an ambiguous acronym and a dictionary with possible expansions (i.e., long-forms) of the acronym. For instance:

Input - Sentence: They use CNN in the proposed model.

Input - Dictionary: CNN: 1. Convolutional Neural Network, 2. Cable News Network

Output: Convolutional Neural Network

In this example, the ambiguous acronym in the input sentence is shown in boldface and the expected prediction for its correct meaning is "Convolutional Neural Network". For this task, participants are provided with the training and development datasets consisting of 62,441 sentences and a dictionary of 732 ambiguous acronyms. The dataset and dictionary are created from 6,786 English scientific papers published at arXiv. The submitted systems will be evaluated based on their precision, recall and F1 scores on the hidden test set. The corpus and the evaluation scripts can be downloaded from https://github.com/amirveyseh/AAAI-21-SDU-shared-task-2-AD

Participation

In order to participate, please first fill out this form to register for the shared tasks: https://forms.gle/NvnT549mSbyeJQAPA. The team name that is provided in this form will be used in the subsequent submissions and communications. The shared tasks are organized in two separate phases:

• Development Phase: In this phase, the participants will use the training/development sets provided for each task in their corresponding GitHub page to design and develop their models. For each task, please refer to its corresponding GitHub page to download the corpus and the evaluation scripts. The dataset folder in the git repository contains the following files:

• Train.json: The training file for the corresponding task. Please refer to the GitHub pages of the AI and AD tasks for the details.

• Dev.json: The development dataset for the corresponding task. This file has the same format as the "train.json" file. Use this file to evaluate your model in the development phase.

• Predictions.json: This file is the reformatted version of the "dev.json" file to represent the format which is expected by the evaluation scripts. Please format the results of your model predictions in the same format as this file to evaluate the model performance using the provided evaluation scripts.

For more details on the dataset and the evaluation scripts, please refer to the AI and AD GitHub Page.

• Evaluation Phase: Two weeks before the system runs due, i.e., 20th November 2020, the test sets for each tasks are released in their corresponding GitHub pages. The test sets have the same distribution and format as the development sets. Run your model on the provided test sets and save the prediction results in a Json file with the same format as the "predictions.json" file. Name the prediction file as "output.json" and send that to the email address sdu-aaai21@googlegroups.com with title "Results of [TASK]-[TEAM-name]-[RUN-ID]", where "[TASK]" should be replaced with either "AI", for acronym identification task, or "AD", for acronym disambiguation task; "[TEAM-name]" with the name of your team provided in the registration form and "[RUN-ID]" with a number between 1 to 10 to identify the model run. Each participant team is allowed to submit up to 10 different model runs. Note that your official score is reported for the model run with ID 1. In addition to the "output.json" file, please include the following information in your email:

  • Model Description: A brief summary of the model architecture. If your model is using word embedding, please specify what type of word embedding your model is using.

  • Extra Data: Whether or not the model employs other resources/data, e.g., acronym glossaries, in the development or evaluation phases.

  • Training/Evaluation Time: How long the model takes to be trained/evaluated on the provided dataset

  • Run Description: A brief description on what is the difference in the recent model run compared to other runs (if it is applicable)

  • Plan for System Paper: If you have any plan to submit your system paper or release your model publicly, please specify that.

The winner of both tasks and selected teams (based on the substance of the approach) will be invited to orally present their system in the workshop. The other participants are invited to present their models in the poster session.

*Update*: The CodaLab competitions for both shared tasks are open. Participants can also submit their results to Acronym Identification and Acronym Disambiguation competitions. For more information, please check the CodaLab competition for Acronym Identification and Acronym Disambiguation.

System Papers

The participants of the both tasks are encouraged to submit their system papers to SDU@AAA-21 workshop. The system papers should provide details of the model architecture, the training method and all resources employed by the model in the training/evaluation phase, and the analysis of the strengths/weaknesses of the proposed model. Please use AAAI 2020 author kit (Note that the 2021 author kit should not be used, as it has a permanent AAAI copyright slug, which is not appropriate for workshops) to write the system papers following the AAAI 2021 formatting guidelines. System papers are limited to 8 pages, including the references. The papers will receive feedback from the workshop program committee and will be published in the workshop proceedings under the shared task section.

Please submit the PDF formatted system papers to EasyChair.

Important Dates

• Training and development set release: September, 1, 2020

• Test set release: November 20, 2020

• System runs due date: December 4, 2020

• System papers due date: December 11, 2020

• Presentation at SDU@AAAI-21: February 9, 2021