Gender Rewriting
Shared Task

Shared Task Description

Data: All participating teams will use the publicly available Arabic Parallel Gender Corpus v2.1 to train and test their systems. Participants are not allowed to use external manually labeled datasets, but they can leverage unlabeled data to create synthetic examples (i.e., data augmentation). A blind test set will be used to evaluate the outputs of participating teams. All teams are required to report on the development and test sets in their write ups.

The blind test is available and can be obtained through this link. We provide 7,318 sentences (52,702 words).

Metrics: We will treat the task of gender rewriting as a user-aware grammatical error correction task and use the M2 Scorer as the evaluation metric. The M2 Scorer computes the Precision, Recall, and F0.5 of the word-level edits between the input and the rewritten output against the gold edits.

We provide an evaluation script for participants so they can start developing their systems. Details on running the evaluation script can be found in this repo.

When you submit your system outputs on the test set, we expect to receive four different files. Each one of those files should contain the gender rewritten alternatives of the sentences in the test set based on the four different target gender contexts and should be named accordingly:

1) test.ar.MM: contains the speaker masculine -- listener masculine gender rewritten alternatives.

2) test.ar.MF: contains the speaker masculine -- listener feminine gender rewritten alternatives.

3) test.ar.FM: contains the speaker feminine -- listener masculine gender rewritten alternatives.

4) test.ar.FF: contains the speaker feminine -- listener feminine gender rewritten alternatives.

Once you have all four files, put them in a zip folder and email them to Bashar: alhafni@nyu.edu

Registration: To participate in the shared task, you must register through this link.

Important Dates

• June 15, 2022: Shared task announcement. Release of training and development data, and evaluation script.

• UPDATED: August 14, 2022 (August 7, 2022): Registration deadline.

• August 8, 2022: Release of test data. Test data is now available here.

• UPDATED: September 15, 2022 (September 12, 2022): Deadline for system output collection.

• UPDATED: September 15, 2022 (September 14, 2022): System description paper submissions due.

• October 10, 2022: Notification of acceptance.

• October 21, 2022: Camera-ready versions due.

• December 7, 2022: WANLP 2022 workshop at EMNLP in Abu Dhabi.

All deadlines are 11:59 pm UTC -12h (“anywhere on Earth”).

Shared Task Paper Submission

Please check the paper submission guidelines.

Ethical Considerations

Our underlying intention in organizing this shared task is to increase the inclusiveness of NLP applications that deal with gender-marking morphologically rich languages. Our goal is to encourage NLP researchers to develop gender rewriting models that are aimed at empowering and allowing users to interact with NLP technology in a way that is consistent with their social identities.

We acknowledge that by limiting the choice of gender expressions to the grammatical ender choices in Arabic, we exclude other alternatives such as non-binary gender or no-gender expressions. However, we are not aware of any sociolinguistics published research that discusses such alternatives for Arabic. We stress on the importance of adapting Arabic NLP models to new gender alternative forms as they emerge as part of the language usage.

Contact

For any questions related to this task, feel free to post them on our Slack workspace or on this Google Group. You are also welcome to contact the organizers directly at this email address: gender.rewriting.organizers@gmail.com.