Efficient Online Reinforcement Learning with Offline Data

Question

We have increasing access to prior offline data containing experiences from a behavior policy.

How can we incorporate this prior data into online RL simply + effectively? 🤔

Existing Approaches

Prior work focuses on offline pre-training, requiring additional complexity and design choices.

Problem #1: Value Divergence

Naïvely initializing the replay buffer of Off-Policy RL with offline data results in value divergence:

In offline RL, penalizing OOD data prevents extrapolation errors, fixing this divergence. BUT, this is a form of anti-exploration; harmful when transitioning online!

Enter LayerNorm! LayerNorm bounds extrapolation, but does not load the critic with negative values!

Problem #2: Data Ingestion

But without pre-training, how do we incorporate offline data?

Two key steps:

1: Symmetric sampling of offline and online data 50:50 per batch

2: Increase gradient steps per timestep to ‘backup’ offline data quickly