Embedded Files
for Temporally Coordinated Exploration in Reinforcement Learning
for Temporally Coordinated Exploration in Reinforcement Learning
Haichao Zhang Wei Xu Haonan Yu
ICLR 2022 Spotlight
Abstract
Standard model-free reinforcement learning algorithms optimize a policy that generates the action to be taken in the current time step in order to maximize expected future return. While flexible, it faces difficulties arising from the inefficient exploration due to its single step nature. In this work, we present Generative Planning method (GPM), which can generate actions not only for the current step, but also for a number of future steps (thus termed as generative planning). This brings several benefits to GPM. Firstly, since GPM is trained by maximizing value, the plans generated from it can be regarded as intentional action sequences for reaching high value regions. GPM can therefore leverage its generated multi-step plans for temporally coordinated exploration towards high value regions, which is potentially more effective than a sequence of actions generated by perturbing each action at single step level, whose consistent movement decays exponentially with the number of exploration steps. Secondly, starting from a crude initial plan generator, GPM can refine it to be adaptive to the task, which, in return, benefits future explorations. This is potentially more effective than commonly used action-repeat strategy, which is non-adaptive in its form of plans. Additionally, since the multi-step plan can be interpreted as the intent of the agent from now to a span of time period into the future, it offers a more informative and intuitive signal for interpretation. Experiments are conducted on several benchmark environments and the results demonstrated its effectiveness compared with several baseline methods.
Generative Planning Framework.
GPM has the following three main components.- Plan generation. At each time step, the plan generator generates a plan based on the state.
- Plan update. The old plan is adjusted by one step ``time-forwarding'' of the previous actual plan.
- Plan switch. The replanning signal is generated based on the values of the previous plan and the new one, and only switches if certain criteria is satisfied.
The plan enclosed with a black box indicates the one been selected at that time step. The primitive action to be executed is derived from the plan and is sent to the environment for execution.
Related Publications and Resources
Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning
Haichao Zhang, Wei Xu and Haonan Yu
International Conference on Learning Representations (ICLR), 2022 Spotlight
[arXiv] [OpenReview] [Slides] [Poster] [Code]
Page updated
Google Sites
Report abuse