Embedded Files
This page presents illustrative experiments demonstrating how updating the base policy with a randomly initialized critic function Q(s, a) results in significant deviations from its original trajectory.
In summary, these experiments suggest that fine-tuning the base policy with a randomly initialized critic can lead to unlearning. Once unlearning occurs, it becomes very hard to relearn the policy since it cannot get the sparse reward signal anymore.
0_update.mp4In the StackCube task, a robot arm must pick up a red cube and stack it on a green cube. Initially, a pre-trained base policy (Behavior Transformer) successfully grasps the red cube and accurately places it on the green cube.
100_update.mp4After fine-tuning the base policy with a randomly initialized critic for 100 gradient steps, the policy begins to deviate slightly from the original trajectory. While still able to grasp the red cube, it fails to precisely place it on the green cube.
200_update.mp4Following an additional 100 updates (200 total), the base policy deviates further from the original trajectory, struggling to effectively grasp the red cube
Google Sites
Report abuse