Shared Task
About the Shared Task
The frequent use of acronyms, which are condensed variants of long technical terms, is a barrier to reading scholarly writing. They are common not only in English but also in other languages. Since most of the existing methods for acronym understanding are dedicated to English text, in this shared task we encourage research on acronym understanding in other languages. Specifically, in addition to English, we provide resources in French, Spanish, Danish, Persian, and Vietnamese. A document reading system should be able to recognize acronyms and their correct meanings in order to process a document correctly in these languages. Because acronyms can be defined both locally within a document and globally in an external dictionary with numerous meanings, a good document comprehension model must capture both local definitions and disambiguate acronyms that are not defined in documents. To push forward the research on acronym understanding, we propose two shared tasks at SDU@AAAI-22.
Task 1 - Acronym Extraction
This task aims to identify acronyms (i.e., short-forms) and their meanings (i.e., long-forms) from the documents. For instance:
Input: Existing methods for learning with noisy labels (LNL) primarily take a loss correction approach.
Output: Existing methods for learning with noisy labels (LNL) primarily take a loss correction approach.
In this example, the acronym is shown in boldface and the long-form is shown with an underline. This task is modeled as a paragraph-level sequence labeling problem. Participants are provided with manually labeled training and development datasets consisting of 4,000 English, 1,000 Persian, and 800 Vietnamese paragraphs in the scientific domain and 4,000 English, 8,000 French, 6,400 Spanish, and 3,000 Danish paragraphs in the legal domain. The submitted systems will be evaluated based on their precision, recall, and F1 scores on the hidden test set computed for correct predictions of short-form (i.e., acronym) and long-form (i.e., phrase) boundaries in the sentences. The corpus and evaluation scripts can be downloaded from https://github.com/amirveyseh/AAAI-22-SDU-shared-task-1-AE
Task 2 - Acronym Disambiguation
This task aims to find the correct meaning of an ambiguous acronym in a given sentence. The input to the system is a sentence containing an ambiguous acronym. The systems are expected to find the correct expanded form of the acronym given the possible expansions for the acronym. For instance:
Input Sentence: All systems use their IP address to introduce themselves to the network.
Input Candidate Long-forms:
Internet Protocol
Intellectual Property
Output: Internet Protocol
In this example, the ambiguous acronym in the input sentence is shown in boldface. In this task, there is no overlap between the acronyms in the training set and the evaluation set. For this task, participants are provided with the training and development datasets in English (both scientific and legal domain), Spanish, and French consisting of 457 English Scientific, 273 English legal, 493 Spanish, and 609 French acronyms. On average, each acronym has 3.1 long-forms. Each language split has its own test set with acronyms not appearing in their training set. The submitted systems will be evaluated based on their precision, recall and F1 scores on the hidden test set. The corpus and the evaluation scripts can be downloaded from https://github.com/amirveyseh/AAAI-22-SDU-shared-task-2-AD
Participation
In order to participate, please first fill out this form to register for the shared tasks: https://forms.gle/njVArce6cwgFmZjG7. The team name that is provided in this form will be used in the subsequent submissions and communications.
The shared-task will be conducted in CodaLab competitions. Participants can submit their results to Acronym Extraction and Acronym Disambiguation competitions. For more information, please check the CodaLab competition for Acronym Extraction and Acronym Disambiguation.
The shared tasks are organized in two separate phases:
Development Phase: In this phase, the participants will use the training/development sets provided for each task in their corresponding GitHub page to design and develop their models. For each task, please refer to its corresponding GitHub page to download the corpus and the evaluation scripts. The dataset folder in the git repository contains the following files:
Train.json: The training file for the corresponding task. Please refer to the GitHub pages of the AE and AD tasks for the details.
Dev.json: The development dataset for the corresponding task. This file has the same format as the "train.json" file. Use this file to evaluate your model in the development phase.
For more details on the dataset and the evaluation scripts, please refer to the AE and AD GitHub Page.
Evaluation Phase: 10 days before the system runs due, i.e., 1st November 2021, the test sets for each task are released in their corresponding GitHub pages. The test sets have the same distribution and format as the development sets. Run your model on the provided test sets and save the prediction results in a JSON file with the same format as the "predictions.json" file. Name the prediction file as "output.json" and submit it to the CodaLab competition page. In addition to the "output.json" file, please include the following information in your submission:
Model Description: A brief summary of the model architecture. If your model is using word embedding, please specify what type of word embedding your model is using.
Extra Data: Whether or not the model employs other resources/data, e.g., acronym glossaries, in the development or evaluation phases.
Training/Evaluation Time: How long the model takes to be trained/evaluated on the provided dataset
Run Description: A brief description on what is the difference in the recent model run compared to other runs (if it is applicable)
Plan for System Paper: If you have any plan to submit your system paper or release your model publicly, please specify that.
The winner of both tasks and selected teams (based on the substance of the approach) will be invited to orally present their system in the workshop. At least one author of accepted papers should register at the conference and present the work at the workshop.
System Papers
The participants of the both tasks are encouraged to submit their system papers to SDU@AAA-22 workshop. The system papers should provide details of the model architecture, the training method and all resources employed by the model in the training/evaluation phase, and the analysis of the strengths/weaknesses of the proposed model. Please use AAAI 2022 author kit to write the system papers following the AAAI 2022 formatting guidelines. System papers are limited to 8 pages, including the references. The papers will receive feedback from the workshop program committee and will be published in the workshop proceedings under the shared task section.
At least one author of accepted papers should register at the conference and present the work at the workshop.
Please submit the PDF formatted system papers to EasyChair.
Important Dates
Training and development set release: September 10, 2021
Test set release: November 1, 2021
System runs due date: November 10, 2021
System papers due date:
November20, 2021November 24, 2021Presentation at SDU@AAAI-22: March 1, 2022