Action Representations for Learning in Continuous Control

Half-day workshop at RSS 2020

July 13, 2020

Virtual Workshop

Update: Video recordings are available here.

Update:

Following the official recommendations, RSS and its workshops will take place virtually. Our workshop will be held at the same day and time (July 13, morning) in a virtual manner, keeping the discussions and presentations live. Contributed papers will be presented as highlight virtual presentations during the workshop.

Abstract

When humans learn new tasks, they connect sensory inputs to actions to create sensory-motor loops. These loops are not always best learned in the same action space. While some tasks are easy to learn and control in joint space, others are easier to define in Cartesian or other task spaces. Even more, some tasks require not only to learn motion but also other dynamic components like the amount of force to apply on the environment.

In robot learning, for example in reinforcement learning, the goal is also to learn sensorimotor loops with a policy that maps observations to actions, elements of some action space. Given that many robotic platforms take desired joint torques as robot commands, a possibility is to learn directly a map ping from sensor signals to torques (i.e. “from pixels to torques”). However, this setup adds complexity to the learning problem since the task can be more easily learned in a different action space.

Recently, several efforts explore how to compose analytical controllers into robot learning frameworks as a way to factorize the learning problem into policy learning and control. The policy still learns to map observations to actions that are reference signals to the controller; a task where methods like Deep RL have succeeded even using high dimensional inputs images.

These actions are mapped into low level robot commands by the analytic controller. The controller, thus, enables a different action space for robot learning. These recent studies confirmed the intuition that different action spaces facilitate learning, depending on the characteristics of the task to learn.

In this workshop we propose to study further this new paradigm that bridges robot learning and advanced robot control. We will try to shed light on questions such as: what properties of manipulation tasks indicate the best action space for robot learning? How to best incorporate compliant and force control into learning algorithms? Do the most common action spaces emerge as natural representation to learn tasks?

Call for contributions

We invite all authors with work relevant to the topic to submit a four-page abstract (using RSS format) June 8, 2020 through the CMT system here. The abstracts will be reviewed in a single-blind process.

Authors will be notified of acceptance by June 15, 2020. Selected contributions will be highlighted as short spotlight presentations during the workshop.

Schedule

All times are PST:

08:00 AM - 08:20 AM Introduction by Roberto Martín-Martín (video)

08:20 AM - 08:50 AM Daniel Braun (video)

08:50 AM - 09:20 AM Marc Toussaint (video)

09:20 AM - 09:50 AM Dagmar Sternad (video)

09:50 AM - 10:50 AM Paper spotlight presentations + Q&A (video)

10:50 AM - 11:20 AM Franziska Meier (video)

11:20 AM - 11:50 AM Michiel van de Panne (video)

11:50 AM - 12:30 PM Panel discussion (video)

Speakers

Daniel Braun

University of Ulm

Learning sensorimotor abstractions: an information-theoretic approach

Leading theories of motor control assume that the motor system learns probabilistic models and that motor behavior can be explained as the optimization of payoff or cost criteria under the expectation of these models. Here we discuss how the motor system exploits task variability to build up efficient abstractions through structural learning and how the learning of such abstractions may be guided both by external costs and internal information processing costs. We compare human behavior to such optimal models and discuss in how far sensorimotor abstractions may subserve meta-learning behavior.

Dagmar Sternad

Northeastern University

Control and Learning of Human Interactions with Complex Objects - A Task-Dynamic Approach

Tool use is the basis for our evolutionary advantage, and everyday life is full of actions that involve physical interaction with objects. Manipulation becomes particularly intriguing when the objects have internal degrees of freedom that add complex dynamics, such as transporting a cup of sloshing coffee. Over the last decades human motor neuroscience has made advances in understanding the control of simple movements, most prominently straight-line reaching in a horizontal plane, and has shed light on adaptation and control. While this paradigm affords strict experimental control of human variability, it is difficult to extrapolate insights to more complex movements, especially interactions with objects where bidirectional forces introduce complex dynamics to be exploited or suppressed. However, studying the control of interactions challenges current experimental and theoretical methods. We have developed an approach that is grounded in the analysis of the object’s dynamics in the context of a task goal. Based on minimal assumptions about the human controller, we parse the task performance into execution and result variables that define the solution space with a manifold of solutions to the task goal. Different solution spaces can be compared to identify the most appropriate execution and result variables. Once identified, this space presents a foundation to examine human control and learning. I will demonstrate this approach and findings by examining the task of transporting a “cup of coffee”, a complex object with underactuated dynamics.

Marc Toussaint

TU Berlin

Physical Manipulation Planning & Learning

What should be learned, and what assumed known about our world, is among the core questions of robotics and AI in general. I will first discuss work on physical manipulation planning that fully builds on carefully formulated analytical models of our world. Once we have such purely model-based solvers, they open great opportunities for learning. I will briefly discuss learning to reason much more efficiently, as well as learning to execute plans.

Franziska Meier

Facebook AI Research

Learning Loss Functions for Fast Adaptation of Robot Dynamics

Meta-learning is a promising route towards enabling fast learning of new skills by transferring knowledge from previously learned behaviors. In this talk I will review some of the existing meta-learning approaches with a focus on which actions-space they utilize. I will then highlight the opportunities of learning loss functions for fast adaptation of dynamics models.

Michiel van de Panne

University of British Columbia

Action Spaces for Locomotion

Legged locomotion is a rich continuous control task having many possible action spaces that could be used for control: torques, muscle activations, PD-targets, ground-reaction forces, and learned action abstractions. Which give the best performance? Which are best for learning? Which transfer best to hardware? We describe our past and ongoing work towards these problems, including direct comparisons of learning in various action spaces, and learned solutions to difficult locomotion tasks. Finally, we argue that action spaces are fully intertwined with the larger problem of task abstraction.