We present 🔑 KAGI: Keypoint-based Affordance Guidance for Improvements. 

KAGI consists of two main components:

(1) We leverage a VLM to select from a set of candidate affordance keypoints, then generate a waypoint sequence forming a trajectory towards the goal.

(2) We perform a per timestep reward computation for each frame in the episode replay buffer, computing dense reward with respect to the waypoint sequence and a sparse reward derived from a success classifier. The dense reward is used for online RL if the sparse reward is 0, else the sparse reward is used.

On four real-world manipulation tasks specified by natural language descriptions, KAGI improves the sample efficiency of autonomous RL and enable successful completion of the task in 30K online finetuning steps.

We compare our approach to MOKA, which executes actions planned by a VLM zero-shot. From this, we see that online fine-tuning is advantageous as KAGI learns policies robust to environment perturbations.

Furthermore, we reduce the number of in-domain demonstrations used in the pipeline by 5x. Even with fewer demonstrations, KAGI can recover comparable performance with more fine-tuning. KAGI's dense rewards improve the system's robustness to reduced demonstrations, facilitating scalability to new tasks.

In simulation experiments, we verify that both dense and sparse rewards are critical for task success, and that
KAGI is robust to reductions in quantity of task demonstrations.

Video Summary