Tool use remains a long-standing challenge in robotics: tasks such as slicing, peeling, and mashing require repeated object transformations while adapting to geometry and material properties. Humans acquire such skills by observing others, reusing familiar movement patterns, and refining them through interaction. Inspired by developmental psychology research on repeated motor patterns, we represent these behaviors as reusable motion primitives, factorized into where to act and how to interact. We introduce TOOLCRAFT, which extracts primitives from long-horizon demonstrations, organizes them into a compact skill space, and learns representative skill modes for coarse-to-fine exploration on novel objects. We also introduce TOOLCRAFTBENCH, a physics-based simulation benchmark for repeated tool-mediated object transformations. In simulation, TOOLCRAFT outperforms monolithic trajectory imitation and fixed-primitive baselines, generalizes across action locations and novel tool geometries, and adapts to unseen objects with few interactions. Real-robot experiments further demonstrate that imperfect primitives recovered from human videos can initialize the skill space and be refined through interaction. These results establish reusable, learned tool-use primitives as a data-efficient representation for object state change tasks.
Extract repeated tool–object interactions from long-horizon demonstrations into a compact skill space. At deployment, explore, refine, and reuse representative skill modes across locations on seen and novel objects.
a) We decompose long-horizon demonstrations into periodic contact primitives that capture reusable tool-object interactions.
b) Each primitive is factorized into a spatial component—where to act—and a learned primitive manifold—how to interact; smooth motion augmentation enriches this manifold from sparse demonstrations.
c) Visual grounding predicts the spatial target from the full scene and the primitive latent from an object-centric crop.
d) At test time, TOOLCRAFT enables sample-efficient exploration in the learned latent space: unsupervised skill modes provide executable anchors for coarse-to-fine refinement, recovering effective interaction strategies with only a few physical trials.
We introduce TOOLCRAFTBENCH, a physics-based simulation benchmark that captures persistent, contact-driven object state change across three tool-use tasks: mashing, peeling, and slicing. Persistent state changes make progress cumulative and interactions history-dependent.
The object is modeled as a spatial height field with persistent deformation and damage. Normal loading compresses the object, while harder materials require additional tangential or torsional work to yield effectively. A simple press may suffice for soft objects, whereas harder ones benefit from rocking or twisting. Continued interaction after sufficient deformation accumulates irreversible damage, distinguishing effective mashing from indiscriminate force.
Peeling models progressive surface-layer removal under constrained blade contact. Removal occurs when the tool maintains appropriate contact and tangential motion, exposing connected regions along its path. Excessive loading causes persistent gouging and alters subsequent interactions, making success depend on both surface coverage and retained object quality.
Progressive material separation is modeled along the observed blade path. Blade-object contact accumulates a material-dependent cutting dose from normal loading and tangential blade motion; once sufficient progress spans the object cross-section, the resulting kerf persists as a separating seam. Harder materials therefore favor motions that generate sustained cutting work, such as sawing, while incomplete or excessive interaction leads to partial cuts or quality loss.
TOOLCRAFT substantially outperforms monolithic BC and fixed-primitive baselines on seen and held-out object-location pairs
TOOLCRAFT substantially outperforms the baselines on unseen tool geometries across objects
TOOLCRAFT performance is robust across changes in tool size, shape, and morphology;
green outlines mark the canonical training geometry.
TOOLCRAFT's cluster-guided latent search discovers effective primitives substantially faster than
raw-trajectory, behavior-cloning, and spatial-search baselines.
TOOLCRAFT first explores skill modes learnt from human videos, then refines and reuses most promising mode across the object