Abstract

Leveraging sensing modalities across diverse spatial and temporal resolutions can improve performance of robotic manipulation tasks. Multi-spatial resolution sensing provides hierarchical information captured at different spatial scales and enables both coarse and precise motions. Simultaneously multi-temporal resolution sensing enables the agent to exhibit high reactivity and real-time control. In this work, we propose a framework for learning generalizable language-conditioned multi-task policies that utilize sensing at different spatial and temporal resolutions using networks of varying capacities to effectively perform real time control of precise and reactive tasks. We leverage off-the-shelf pretrained vision-language models to learn generalizable multi-task policies along with small non-pretrained models to adapt to high resolution feedback. Through extensive experiments in 3 domains (coarse, precise and dynamic manipulation tasks), we show that our approach significantly improves (2x on average) over recent multi-task baselines. Further, our approach generalizes well to visual and geometric variations in target objects and to varying interaction forces.

Real World Results

Real World Low Temporal-Resolution Insertion

As can be seen in the following qualitative results, a low temporal-resolution policy (i.e. where all input modalities are processed at 5Hz), does not lead to a very reactive policy. Hence, the policy is unable to respond fast to contacts made with the wooden peg. Since completing this task requires the policy to find (approximately localize) the insertion location, a slow to react policy is often unable to find the insertion location with the time horizon.

Dynamic Instruction Following

Task 1:  Initial Instruction: Lift the blue shoe. New Instruction: Lift the green shoe.

Task2:  Initial Instruction: Reach above red block. New Instruction: Reach above green block.