Non-Parametric Stochastic Policy Gradient with Strategic Retreat for Non-Stationary Environment
Apan Dastider and Mingjie Lin
Autonomous Computing Lab, University of Central Florida
Arxiv Paper Link : https://arxiv.org/abs/2203.14905
Embedded Files
Appendix
Appendix
complete_writeup_rkhs (9).pdfMethodology of tuning adaptive H window for optimal actions based on dot product based similarity metric in RKHS and revised new policy after Adaptive Retreat :
Methodology of tuning adaptive H window for optimal actions based on dot product based similarity metric in RKHS and revised new policy after Adaptive Retreat :
Abstract --- In modern robotics, effectively computing optimal control policies under dynamically varying environments poses substantial challenges to the off-the-shelf parametric policy gradient methods, such as the Deep Deterministic Policy Gradient (DDPG) and Twin Delayed Deep Deterministic policy gradient (TD3). In this paper, we propose a systematic methodology to dynamically learn a sequence of optimal control policies non-parametrically, while autonomously adapting with the constantly changing environment dynamics. Specifically, our non-parametric kernel-based methodology embeds a policy distribution as the features in a non-decreasing Euclidean space, therefore allowing its search space to be defined as a very high (possible infinite) dimensional RKHS (Reproducing Kernel Hilbert Space). Moreover, by leveraging the similarity metric computed in RKHS, we augmented our non-parametric learning with the technique of AdaptiveH— adaptively selecting a time-frame window of finishing the optimal part of whole action-sequence sampled on some preceding observed state. To validate our proposed approach, we conducted extensive experiments with multiple classic benchmarks and one simulated robotics benchmark equipped with dynamically changing environments. Overall, our methodology has outperformed the well-established DDPG and TD3 methodology by a sizeable margin in terms of learning performance.
Non-Stationary Environments and Learning Curves:
Non-Stationary Environments and Learning Curves:
Real Hardware Experiments:
Real Hardware Experiments:
Evolution of H
Evolution of H
Handling Varying NS
Handling Varying NS
A Video Demonstration of our real-hardware experimentation on Franka Emika Panda 7-DOF Robotic Manipulator and an animated action selection procedures based on Adapive Retreat
A Video Demonstration of our real-hardware experimentation on Franka Emika Panda 7-DOF Robotic Manipulator and an animated action selection procedures based on Adapive Retreat
Page updated
Google Sites
Report abuse