What are valuable benchmarks for language emergence?

Luc Steels, ICREA, UPF Barcelona, Spain

An agent-based model of language emergence proposes an environment, a population of agents with a specific cognitive architecture, and a script for a language game that achieves a particular communicative goal. The model is tested by letting the agents play language games in order to see whether the population indeed collectively self-organizes a communication system adapted to the environment and to the communicative goal. To be of interest and relevant to the study of human language emergence, the communication system should exhibit some key characteristic of human language, such as the expression of rich conceptualizations or hierarchical syntax.

Benchmarks for the emergence of grounded spatial language

Michael Spranger, Sony Computer Science Laboratory, Tokyo, Japan

This talk introduces concrete benchmarks for studying the emergence of spatial language, i.e. spatial terms (e.g. "left", "next-to", "on top of"), spatial expressions (e.g. "left of the box"), and spatial perspective (e.g. "behind the box from your perspective", "to your right"). The benchmarks include data drawn from physical robot interaction in a blocks world environment. They allow us to study the emergence of perceptual categories, grounded spatial relations, item-based syntax, argument structure, and compositional procedural semantics. I will also introduce agent-based models that successfully handle the benchmarks and introduce an additional benchmark involving events to study temporal language constructs.

Benchmarks for exploring the relationship between theory of mind and emergent communication

Jakob Foerster, Oxford University, UK

Humans routinely make inferences about the state of mind, motivations, and beliefs of others when observing their actions. This ability, typically referred to as "theory of mind", is believed to play an important role in the discovery of communication protocols and the learning of language. In this talk we present a benchmark which is well suited for exploring this connection and for measuring our progress as a field. While benchmarks have driven progress in machine learning and artificial intelligence over the last few decades, the field of emergent communication and language is urgent need of more standardized testbeds for measuring progress. I believe that the benchmark presented in this talk will be part of the solution.

Benchmarks for studying the emergence of syntactic structure

Freek Van de Velde, Linguistics Department, University of Louvain-KUL, Belgium

Can we use insights and data from historical linguistics to create benchmarks for agentbased models of language emergence? Here I will focus on the emergence of noun-phrase constituency in West-Germanic and Romance using historical demographic databases we have collected for a burgeoning project on demography-driven morphosyntactic change. I will show how these data allow us to investigate the dynamic nature of hierarchical structure in language and the trade-off between morphological complexity and syntactic/lexical constructions. I will also introduce reliable metrics for morpho-syntactic complexity to track in how far agent-based models achieve phenomena observed in the emergence and evolution of grammar.

Benchmarks for studying the effect of perceptual constraints and socio-cultural transmission on the emergence of color terms

Tao Gong, ETS, New Jersey, USA

The origins of color terms and color expressions has been an important target for agent-based models, because it is relatively easy to collect sensory data and extensive surveys of color word usage are available starting with the seminal work of Berlin and Kay. Here I focus on agent-based models that examine the impact of two factors influencing color language: perceptual constraints and the socio-cultural environment in which agents can communicate using color language. In this way, we can evaluate the relative importance between individual perceptual constraints and socio-cultural transmissions on shaping the color categorization patterns evident in the World Color Survey.

Sponsoring Institutions