IJCAI'15 Tutorial on Evolutionary Semantics for Language Grounding in Robots

The tutorial focuses on how robots could make use of sophisticated compositional semantics that is grounded in their sensori-motor systems and evolves with changing contexts and goals. It surveys the issues to be solved, past experiments, and current techniques, illustrated with concrete case studies.

The tutorial is part of the International Joint Conference on Artificial Intelligence (IJCAI), at Buenos Aires, Argentina, July 25-31, 2015.

Date: Sunday July 26th

Time: 8:45 - 12:45

Place: New Building of Facultad de Ciencias Económicas, Room 465

Please also check the tutorial schedule page of IJCAI

Description

Natural language interaction between humans and robots (or more broadly autonomous intelligent systems such as self-driving cars) remains one of the biggest challenges of AI, mainly because it requires integration of highly sophisticated components for vision and motor control, speech, parsing and production of language, interaction through dialog, and grounded semantics. All these components should ideally acquire content through machine learning and have to remain adaptive to changing contexts, goals and interlocutors. This tutorial focuses on how to achieve evolutionary grounded semantics.

The tutorial first surveys the prior art, in particular Shakey and Shrdlu, declarative semantics, including the use of PROLOG-based robot languages, simulation semantics, frame semantics and procedural semantics. It also introduces basic notions of (cultural) evolution and how they could be used to make semantics self-generated and adaptive. The tutorial then zooms in on procedural semantics. Procedural semantics sees meaning in terms of procedures operating over sensori-motor states and world models. The tutorial examines possible representational languages for procedural semantics, how the process of conceptualization can be seen as a planning problem, how concepts required for conceptualization get learned, and how lexicons and construction grammars are able to express procedural semantics. To illustrate the main points at a technical level, the tutorial uses case studies in a number of different domains, in particular, reference to objects based on their properties, producing and understanding action commands, and spatial and temporal description of situations.

Schedule and Materials

The tutorial is organized into 5 modules

• INTRODUCTION. Overview of the steps involved in speaking and comprehending. Examples of historic integrated systems (Shakey, Shrdlu) and contemporary platforms for building natural language interfaces (e.g. TUM-Rosie, PR2, NAO, QRIO). Introduction to evolutionary semiotic dynamics.

• LEXICAL LANGUAGES. Introduction to grounded language games such as the grounded naming game and the connection with sensorimotor systems. Self-organization of vocabulary and adaptive concept evolution.

• GRAMMATICAL LANGUAGES. Introduction to construction grammar. Examples from the emergence of phrase structure grammar. Comparison of different grammatical strategies, emergence of markers and syntactic categories.

• GROUNDED SEMANTICS. Introduction to compositional grounded semantics. Case studies of representing locative and dynamic spatial language using procedural grounded networks

• LEARNING AND EVOLUTION. Learning concepts for grounded conceptual semantics. Configuring and chunking of procedural semantics networks. How such networks get shared between agents. Evolution and adaptation to new domain and goal challenges for situated interactions.

Detailed Schedule

• 08:45 to 10:30 - Part I: Introduction, lexical and grammatical languages

• 10:30 to 11:00 - Coffee break

• 11:00 to 12:45 - Part II: Grounded semantics, learning and evolution

Literature

We recommend the following literature for the tutorial

Books

• Steels, Luc, and Manfred Hild, eds. Language grounding in robots. Springer Science & Business Media, 2012.

• Steels, Luc, ed. Experiments in cultural language evolution. John Benjamins Publishing, 2012.

• Steels, Luc The Talking Heads experiment: Origins of words and meanings. Language Science Press, 2015

Papers

• • Spranger, Michael, Simon Pauw, Martin Loetzsch, and Luc Steels. Open-ended Procedural Semantics. In L. Steels and M. Hild, editors, Language Grounding in Robots, pages 153-172. Springer, 2012.

  • Michael Spranger and Simon Pauw. Dealing with Perceptual Deviation - Vague Semantics for Spatial Language and Quantification. In L. Steels and M. Hild, editors, Language Grounding in Robots, pages 173-192. Springer, 2012.

  • Spranger, Michael. Evolving grounded spatial language strategies. KI - Künstliche Intelligenz, 27(2):97-106, 2013.

Software

The tutorial introduces and discuss software frameworks such as Incremental Recruitment Language (IRL) and Fluid Construction Grammar (FCG). Interested participants are invited to download and try out the software and accompanying examples. Please use the following link to download and install Babel2 - a software package that includes FCG and IRL: Babel2 download.

Target Audience

The target audience consists of researchers in robotics and AI, who are interested to work on the interface of the two fields as a way to advance the state of the art in both. No prior in-depth knowledge of robotics, semantics or language processing is required. The primary goal of the tutorial is to make participants aware of the issues involved in language grounding (which are usually underestimated) and to give a clear idea of the current state of the art and the open problems.

Organizers

Luc Steels, Institute for Advanced Studies (ICREA), Barcelona, Spain

Michael Spranger, Sony Computer Science Laboratories Inc. Tokyo, Japan