Tool as Embodiment for Recursive Manipulation

• Hand-object interaction and tool-object interaction can have similar dynamics but prior work often deals with them using separate approaches

• We propose one way to unify them and train a single neural network policy to do both (e.g. pick up tools and use them to pick up an object)

• Results show models trained on both types of interactions may be beneficial for generalization

• Tool use may be one path to train robot policies that are robust to changes in embodiment

The architecture we use is similar to that of Transporter or AdaGrasp, but the output is conditioned on the scene and an end effector representation that can capture both grippers and tools.

We created 4 tasks spanning both hand-object interaction and tool-object interaction:

Results indicate that the learned model can beat baselines and that a model trained on both hand-object interaction and tool-object interaction can improve performance over one trained separately. However, there is still room for improvement and testing in more complex settings.