RepEval 2016

The First Workshop on Evaluating Vector Space Representations for NLP

12th August 2016, Berlin, Germany

Mission Statement: To develop new and improved ways of measuring the quality or understanding the properties of vector-space representations in NLP.


Models that learn real-valued vector representations of words, phrases, sentences, and even documents are ubiquitous in today's NLP landscape. These representations are usually obtained by training a model on large amounts of unlabeled data, and then employed in NLP tasks and downstream applications. While such representations should ideally be evaluated according to their value in these applications, doing so is laborious, and it can be hard to rigorously isolate the effects of different representations for comparison.

The Challenge

There is therefore an important need for better ways to evaluate semantic representations of language via simple and generalizable proxy tasks. To date, these proxy tasks have been mainly focused on lexical similarity and relatedness, and do not capture the full spectrum of interesting linguistic properties that are useful for downstream applications. This workshop challenges its participants to propose methods and/or design benchmarks for evaluating the next generation of vector space representations, for presentation and detailed discussion at the event. Following the workshop, the highest-quality proposals will receive the support of the organizers and participants, and some financial support, to help produce their proposed resource to the highest standard.

Related Workshop

Prospective attendees of RepEval may be interested in the following companion workshop, also collocated with ACL: the 1st Workshop on Representation Learning for NLP, organised by Phil Blunsom, Kyunghyun Cho, Shay Cohen, Edward Grefenstette, Karl Moritz Hermann, Laura Rimell, Jason Weston and Scott Yih. We expect the topics covered by both workshops to complement one another well, and encourage ACL attendees to consider attending both.