Technical terminology expressions are unique to a particular field and have implied meanings that do not conform to common expectations. Non-experienced individuals would either be completely unaware of the meaning of such concepts or tend to interpret common words in a different sense than intended, failing to understand the communication attempt in either case. This, in turn, creates a significant entry barrier to reading scholarly writing. A primordial first step to creating a scientific-document reading system is to be able to recognize these technical terms that could delay, or even impede, reader understanding. As very few research efforts have been devoted to this matter, we propose a technical terminology detection shared task at SDU@AAAI-23 in an attempt to push forward the research on technical terminology understanding.
This task aims to identify domain-specific technical terminology, a.k.a. jargon, used in scientific research papers. This includes both words that are specific to a field or commonly used terms that acquire particular meanings within such a field. For instance:
Input: Afterwards, model training is performed using cross-entropy loss.
Output: Afterwards, model training is performed using cross-entropy loss.
In this example, the terms "model training" and "cross-entropy loss" are technical computer science concepts that should be identified. To simplify the task, only the main noun in a nominal phrase should be tagged. The task is modeled as a sentence-level sequence labeling problem. Participants are provided with manually-labeled training and development datasets from three distinct scientific domains: Computer Science, Economics, and Physics. Each scientific domain has its own train/dev/test splits. Considering both the training and development sets, the provided dataset includes 7000+ Computer Science sentences, 6000+ Economics sentences, and 8000+ Physics sentences. The submitted systems will be evaluated based on their precision, recall, and F1 scores on the hidden test set computed for correct prediction boundaries in the sentences. The corpus and evaluation scripts can be downloaded from the following GitHub repo.
In order to participate, please first fill out this form to register for the shared task. The team name that is provided in this form will be used in subsequent submissions and communications.
The shared task will be conducted in a Kaggle competition. For more information and result submission, please check the Kaggle competition website: Jargon/Terminology Detection.
The shared tasks are organized in two separate phases:
Development Phase: In this phase, the participants will use the training/development sets provided by the GitHub repository to design and develop their models. Please refer to the GitHub page to download the corpus and the evaluation scripts. The dataset folder in the git repository contains the following files:
Train.json: The training file for the shared task. Please refer to the repo for additional details.
Dev.json: The development dataset for the shared task. This file has the same format as the "train.json" file. Use this file to evaluate your model in the development phase.
Evaluation Phase: Ten days before the system runs are due (January 4th, 2023), the test set for the task will be released in the GitHub repository. The test set has the same distribution and format as the development set. Run your model on the provided test set and save the prediction results in a JSON file with the same format as the "dev.json" file. Name the prediction file "output.json" and submit it to the Kaggle competition page. In addition to the "output.json" file, please include the following information in your submission:
Model Description: A brief summary of the model architecture. If your model is using word embedding, please specify what type of word embedding your model is using.
Extra Data: Whether or not the model employs other resources/data, e.g., domain-specific glossaries, in the development or evaluation phases.
Training/Evaluation Time: How long it takes for the model to be trained/evaluated on the provided dataset.
Run Description: A brief description of the difference between the most recent model run compared to other runs (if applicable).
Plan for System Paper: If you have any plans to submit your system paper or release your model publicly, please specify that.
The winner of the task and selected teams (based on the substance of the approach) will be invited to orally present their system in the workshop. At least one author of accepted papers should register at the conference and present the work at the workshop.
The participants of the shared task are encouraged to submit their system papers to SDU@AAA-23 workshop. The system papers should provide details of the model architecture, the training method and all resources employed by the model in the training/evaluation phase, and the analysis of the strengths/weaknesses of the proposed model. Please use AAAI 2023 author kit to write the system papers following the AAAI 2023 formatting guidelines. System papers are limited to 8 pages, including the references. The papers will receive feedback from the workshop program committee and will be published in the workshop proceedings under the shared task section.
At least one author of accepted papers should register at the conference and present the work at the workshop.
Please submit the PDF formatted system papers to Microsoft CMT.
Training and development set release: November 5, 2022
Test set release: January 4, 2023
System runs due date: January 13, 2023
System papers due date: January 27, 2023
Presentation at SDU@AAAI-22: February 14, 2023
All deadlines are “anywhere on earth” (UTC-12)