Timetable

Haim Dubossarsky

Title:

A space is worth a thousand words: A new spectral analysis method to evaluate vector space similarity.

Abstract:

Vector-based models represent the meaning of words as numeric vectors, based on the words’ co-occurrence usage statistics as reflected in natural texts. These representations are ubiquitous in everyday language technology applications, and are also the object of scientific inquiry in computational linguistic, social sciences, and other data-driven research domains. Despite significant differences in the architecture of different models (e.g., whether they are static or contextualized word embeddings), all models can be thought of as implementing the distributional hypothesis. Perhaps due to the original theoretical framing of this hypothesis (“You shall know a word by the company it keeps”), word vectors are typically analyzed as separate units, and their potential interactions are thus overlooked. This unnecessarily limits the potential that lies in these representations for both scientific research and language technology applications.

I will present a novel framework that analyzes the entire vector space of a language, rather than focusing on individual vectors. Indeed, when the entire semantic space spanned by these vector representations is analyzed using spectral analysis, new information and language related features emerge. I will present results from cross-lingual transfer learning tasks, which are particularly suitable for the testing of the current framework, since performance in these tasks is impacted by the similarity between the languages at hand (i.e. the assumption of isomorphism between vector spaces). I will present a large-scale study focused on the correlations between similarity scores that were developed and computed for vector spaces and task performance, covering thousands of language pairs and four different tasks: Automatic bilingual lexicon induction (BLI), syntactic parsing, Part-Of-Speech tagging and Machine Translation. I will further introduce several similarity-isomorphism measures between two vector spaces, based on the relevant statistics of their individual spectra. I will empirically show that: (a) similarity scores derived from such spectral isomorphism measures are strongly associated with performance observed in different cross-lingual tasks; (b) these spectral-based measures consistently outperform previous standard isomorphism measures which are computed at the word level, while being computationally more tractable and easier to interpret; (c) these novel similarity-isomorphism measures capture complementary information to linguistic distance measures, and the combination of measures from the two types of measures yields even better results. Overall, these findings make an inroad to a new type of analysis, and demonstrate that richer and unique information lies beyond simple word level analysis.

Ellie Pavlick

Title:

Implementing Symbols and Rules with Neural Networks

Abstract:

Many aspects of human language and reasoning are well explained in terms of symbols and rules. However, state-of-the-art computational models are based on large neural networks which lack explicit symbolic representations of the type frequently used in cognitive theories. One response has been the development of neuro-symbolic models which introduce explicit representations of symbols into neural network architectures or loss functions. In terms of Marr's levels of analysis, such approaches achieve symbolic reasoning at the computational level ("what the system does and why") by introducing symbols and rules at the implementation and algorithmic levels. In this talk, I will consider an alternative: can neural networks (without any explicit symbolic components) nonetheless implement symbolic reasoning at the computational level? I will describe several diagnostic tests of "symbolic" and "rule-governed" behavior and use these tests to analyze neural models of visual and language processing. Our results show that on many counts, neural models appear to encode symbol-like concepts (e.g., conceptual representations that are abstract, systematic, and modular), but not perfectly so. Analysis of the failure cases reveals that future work is needed on methodological tools for analyzing neural networks, as well as refinement of models of hybrid neuro-symbolic reasoning in humans, in order to determine whether neural networks' deviations from the symbolic paradigm are a feature or a bug.