Selected Topics

RL and Control with Exogenous Distractors

The state-space of many RL and control problems contains information on an uncontrollable and time-correlated process, which we refer to as exogenous distractors. For example, visual sensors of an autonomous car may detect irrelevant information on a flock of birds. Nevertheless, we would hope, the complexity of learning to drive would be independent of the complexity of learning the movement of the flock. In this line of work, we aim to study algorithms for this setting that have no or mild dependence on the complexity of the exogenous noise process.

Sample Complexity of MDPs with Latent Context

We focus on a natural class of POMDPs in which exists a hidden context, not directly observed as a part of the state space, that we refer to as latent MDPs (LMDPs). We study several aspects of this problem, (i) we establish a statistical separation between the scenario the hidden context is given at the end of each episode and the scenario in which it is not given, (ii) study several assumptions for which this class of problems is learnable with polynomial sample complexity, and (iii) study a sub-class of LMDPs and study a provably efficient algorithm under no assumptions.

Does Offline Data Improve Online Algorithms?

Motivated by this question, we study linear bandits supplied with offline data in which the feature vectors are only partially observed. We then suggest a simple approach, based on geometric intuition, that can utilize the offline data to improve the performance of the online algorithm.

Sequential Decision Making with a Feedback Budget

A recurring theme in applying sequential decision-making algorithms is the lack of feedback. Indeed, in most applications in which a human supplies the reward information there is some feedback limitation. Motivated by this realization, we formulated the sequential budgeted learning setting in which there is a limitation on the available reward feedback. We study a simple principled approach -- we refer to as confidence-budget matching -- for this setting and applied it in RL and contextual bandit problems. Towards a deeper understanding of the setting, we also studied problem-dependent guarantees for the multi-armed bandit problem in the presence of feedback constraints.

Policy optimization for RL

Motivated by the recent success of policy optimization in RL and safe RL we established convergence guarantees to policy optimization techniques. Specifically, we analyzed the convergence of the TRPO algorithm (Schulmann et al. 2015) and showed it has similar performance guarantees as to the Conservative Policy Iteration algorithm (Kakade and Langford 2002). Additionally, we considered the exploration problem and derived performance guarantees of mirror-descent type of algorithm for MDPs and constraint MDPs.

Lookahead policies: when, why, how?

Lookahead policies are a widely used tool in practical applications for several decades -- from Samuel's checkers play program to Alpha-Go. Motivated by these successes, we studied lookahead policies from several perspectives and addressed when, why and how lookahead policies should be used. Furthermore, we established a close relation between lookahead policies and reward shaping, which allowed us to scale multi-step greedy algorithms to deep RL.