Next Seminar

Offline RL via Feature-Occupancy Gradient Ascent

Authors: Gergely Neu, Nneka Okolo

Abstract: We study offline Reinforcement Learning in large infinite-horizon discounted Markov Decision Processes (MDPs) when the reward and transition models are linearly realizable under a known feature map. Starting from the classic linear-program formulation of the optimal control problem in MDPs, we develop a new algorithm that performs a form of gradient ascent in the space of feature occupancies, defined as the expected feature vectors that can potentially be generated by executing policies in the environment. We show that the resulting simple algorithm satisfies strong computational and sample complexity guarantees, achieved under the least restrictive data coverage assumptions known in the literature. In particular, we show that the sample complexity of our method scales optimally with the desired accuracy level and depends on a weak notion of coverage that only requires the empirical feature covariance matrix to cover a single direction in the feature space (as opposed to covering a full subspace). Additionally, our method can be implemented efficiently without requiring any computational oracles, and requires no prior knowledge of the coverage ratio (or even an upper bound on it), which altogether make it the strongest known algorithm for this setting to date.

Speaker Bio: Nneka is a Norbert Wiener Postdoctoral Fellow at the Institute for Data, Systems, and Society (IDSS) in MIT. Prior to this, she completed her Ph.D. in computer science from Pompeu Fabra University in Barcelona, and holds a Bachelor’s degree in Applied Mathematics from University of Benin in Nigeria. Her current research lies at the intersection of reinforcement learning, stochastic optimization and online learning. Towards applying RL for real-world sequential decision-making, she is particularly interested in developing algorithms for RL in large MDPs (where the number of states is finite but intractably large) with provable time- and sample-complexity guarantees.

How To Join:

• Join the Google group here to get details of this, and future seminars.

• Watch the recording of the seminar later on YouTube.