Summer / Été 2019

Quantiles for bandits (and RL?)

The question of “safe” exploration has received a lot of attention recently in RL, and more broadly there has been a move away from average performance to tail performance. One way to achieve this is to move away from maximizing expected rewards to instead maximize other quantiles of the reward distribution. One distinct advantage of quantiles is the following: while means may not exist or could be infinite especially when rewards are heavy-tailed, quantiles are always well defined and exist for any distribution. In this talk, we will gently introduce quantiles, examine some of their advantages, and demonstrate that it is possible to estimate quantiles of a distribution sequentially over time. We then apply our results to the problem of selecting an arm with the best quantile in a multi-armed bandit framework, proving a state-of-the-art sample complexity bound for a novel allocation strategy. Simulations demonstrate that our method stops with fewer samples than existing methods by a factor of five to fifty. A central open problem is extending our ideas to tabular MDPs and related “distributional" RL settings. This is joint work with Steve Howard, a PhD student at UC Berkeley.

Efficient Reinforcement Learning with Generalized Policy Updates

The combination of reinforcement learning with deep learning has led to several successful applications in recent years. However, the amount of data needed by learning systems of this type still precludes their widespread use. In this talk I will describe a divide-and-conquer approach to reinforcement learning with the potential to make it more data efficient. The basic premise is that complex decision problems can be naturally decomposed into multiple tasks that unfold in sequence or in parallel. By associating each task with a reward function, we can naturally incorporate this problem decomposition into the standard reinforcement learning formalism. The specific way I propose to do so is through a generalization of two fundamental operations in reinforcement learning, policy improvement and policy evaluation. The generalized version of these operations, jointly referred to as "generalized policy updates", extend their standard counterparts from single to multiple tasks. This allows one to leverage the solution of some tasks to speed up the solution of other tasks. I will describe how exactly this can be done and provide some examples illustrating the use of the proposed approach in practice.

Rethinking behavior from an evolutionary perspective

In both AI research and psychological theory, the human brain is usually thought of as an information processing system that encodes and manipulates representations of knowledge to produce plans of action. This view leads to a decomposition of behavior into putative functions such as object recognition, memory, decision-making, action planning, etc., inspiring the search for the neural correlates of these functions and attempts to simulate them in computational models. However, empirical neurophysiological data does not support many of the predictions of these classic subdivisions, consistently showing divergence and broad distribution of putatively unified functions, “mixed representations” that combine sensory, motor, and cognitive variables in single neurons, and a general incompatibility with the conceptual subdivisions posited by psychological theories. In this talk, I will explore the possibility of resynthesizing a different set of functional subdivisions, guided by the growing body of data on the evolutionary process that produced the human brain. The main part of my talk will summarize, in chronological order, the sequence of innovations that appeared in nervous systems along the lineage that leads from the earliest multicellular animals to humans. Along the way, functional subdivisions and elaborations will be introduced in parallel with the neural specializations that made them possible, gradually building up an alternative conceptual taxonomy of brain functions. These functions emphasize mechanisms for real-time interaction with the world, rather than for building explicit knowledge of the world, and the relevant representations emphasize pragmatic outcomes rather than decoding accuracy, mixing variables in just the way seen in real neural data. I will argue that this alternative taxonomy better delineates the real functional pieces into which the human brain is organized, offers a more natural mapping to real neural structures, and provides a better set of constraints for constructing artificial systems aimed at replicating brain function

Unsupervised State Representation Learning in Atari

State representation learning, or the ability to capture latent generative factors of an environment, is crucial for building intelligent agents that can perform a wide variety of tasks. Learning such representations without supervision from rewards is a challenging open problem. We introduce a method that learns state representations by maximizing mutual information across spatially and temporally distinct features of a neural encoder of the observations. We also introduce a new benchmark based on Atari 2600 games where we evaluate representations based on how well they capture the ground truth state variables. We believe this new framework for evaluating representation learning models will be crucial for future representation learning research. Finally, we compare our technique with other state-of-the-art generative and contrastive representation learning methods.

Geometry-based Data Exploration

High-throughput data collection technologies are becoming increasingly common in many fields, especially in biomedical applications involving single cell data (e.g., scRNA-seq and CyTOF). These introduce a rising need for exploratory analysis to reveal and understand hidden structure in the collected (high-dimensional) Big Data. A crucial aspect in such analysis is the separation of intrinsic data geometry from data distribution, as (a) the latter is typically biased by collection artifacts and data availability, and (b) rare subpopulations and sparse transitions between meta-stable states are often of great interest in biomedical data analysis. In this talk, I will show several tools that leverage manifold learning, graph signal processing, and harmonic analysis for biomedical (in particular, genomic/proteomic) data exploration, with emphasis on visualization, data generation/augmentation, and nonlinear feature extraction. A common thread in the presented tools is the construction of a data-driven diffusion geometry that both captures intrinsic structure in data and provides a generalization of Fourier harmonics on it. These, in turn, are used to process data features along the data geometry for denoising and generative purposes. Finally, I will relate this approach to the recently-proposed geometric scattering transform that generalizes Mallat's scattering to non-Euclidean domains, and provides a mathematical framework for theoretical understanding of the emerging field of geometric deep learning.

Some progress on understanding the benefits of distributional reinforcement learning - Marc G. Bellemare

In this talk I will review what we now know about distributional reinforcement learning and how its full benefits are only obtained when combined with nonlinear representations such as deep networks. I will discuss how trying to understanding the good empirical performance of distributional RL has led us to all kinds of exciting results regarding representation learning for RL, and in particular a formulation of optimal representation learning based on the geometric notion of a value function polytope.

Model-based RL via Meta-Model-Free RL - Pieter Abbeel

Model-free RL has seen great asymptotic successes, but sample complexity tends to be high. Model-based RL carries the promise of better sample efficiency, and indeed has shown more data efficient learning, but tends to fall well short of model-free RL in terms of asymptotic performance. In this presentation I will describe a new approach to model-based RL that brings in ideas from domain randomization and meta-model-free RL, resulting in the best of both worlds: fast learning and great asymptotic performance. Our method is evaluated on several mujoco environments (PR2 reacher, swimmer, hopper, ant, swimmer, walker) and is able to learn lego-block placement on a real robot in 10 minutes.