Call for Papers

The First Workshop on Evaluations and Assessments of Neural Conversation Systems (EANCS) co-located with EMNLP 2021, will be held in a hybrid mode on Nov 11, 2021. Our sessions will be online and be broadcasted live to the EMNLP event room at at Punta Cana.

Much progress has been made recently to improve conversation systems and chatbots using neural based deep models. While the architectures of conversation models evolve quickly, evaluation techniques over the years remain unchanged. As a matter of fact, evaluating and assessing conversation models has been a decades long challenge. One part of the difficulty comes from the fact that human dialogues exist in a variety of forms. Existing approaches that compare generated conversations from a neural model with ground truth can easily incur large biases due to the lack of diversity. Another part of the difficulty comes from the interactive nature of conversations, which often requires an agent (usually a real person), to conduct the assessments. Human evaluations, on the other hand, can introduce a large amount of variance and are often impractical on a large scale. A third dimension of evaluation has to do with the fairness and reliability of the models, which has become an increasingly important issue for commercial use of neural based systems. Finally, transferability, transparency and ethical issues in evaluations are among some of the other important topics to explore.

Limitations of evaluation and assessment techniques can not only cause difficulties in comparing conversation models but also delay the developments of advanced learning algorithms. Recognizing the importance and the challenges of this topic, we put together this workshop to explore the domain of dialogue evaluation. In particular, we are interested in contributions in the following areas:

• settings, benchmarks, and metrics for automatic evaluation

• simulated agents for evaluation

• human evaluation metrics

• transferability of evaluation methods

• fairness of conversation models

• reliability of conversation models

• interpretability and transparency of evaluations

• datasets for evaluation

• ethical issues in evaluations

For questions regarding the workshop submissions, please feel free to contact the organizing committee at eancs21-organizers@googlegroups.com

Shared Task

We release a new benchmark dataset that aims at facilitating the research of evaluating neural conversational models. In this dataset, a few deep neural models are assessed by human agents, each with a numerical score indicating the performance of the models. Dialogue evaluation methods are assessed by comparing the predicted scores to the actual human scores. Please refer to our shared task page (https://sites.google.com/view/eancs/shared-task) for details.

Important Dates

EANCS 2021 will adopt dual submission deadlines. Accepted submissions before the early deadline (before Sept 5, 2021) will be eligible (pending the approvals from the authors) to be included into the ACL anthology. Accepted submissions after the early deadline will be included in our workshop arxiv. All the deadlines are valid anywhere around the world.

Early Submission Deadline: Sun., Sept 5, 2021
Early Submission Acceptance Notification: Fri., Sept 17, 2021
Early Submission Camera Ready Deadline: Fri., Oct 1, 2021

Regular Submission Deadline: Fri., Oct, 1, 2021
Regular Submission Acceptance Notification: Fri., Oct 15, 2021
Regular Submission Camera Ready Deadline: Fri., Oct 31, 2021

Submission Site

Submissions should be delivered to our workshop portal through SoftConf using the link below. Contact the organizers (eancs21-organizers@googlegroups.com) should you run into any issues with submitting.

https://www.softconf.com/emnlp2021/EANCS/.

Author Guidelines

We welcome short papers as well as extended abstracts. All submissions should follow EMNLP 2021 templates (https://2021.emnlp.org/call-for-papers/style-and-formatting). Submissions should be either short papers (between 4 and 6 pages, excluding references) or extended abstracts (2 pages, excluding references). We welcome submissions that are work in progress, as well as submissions that are currently under review at other venues

Commitment to Diversity

This workshop is committed to providing equal access and participation without regard to race, color, religion, gender identity, among other things. The organizing committee shares a strong commitment to promote diversity.

Useful Links

Workshop Homepage: https://sites.google.com/view/eancs/home
Call for Papers: https://sites.google.com/view/eancs/call-for-papers
Submission Site: https://www.softconf.com/emnlp2021/EANCS/
Shared Task: https://sites.google.com/view/eancs/shared-task
Contact Workshop Organizers: eancs21-organizers@googlegroups.com