Paper: https://arxiv.org/abs/2503.02738v1
2025 IEEE International Conference on Robotics and Automation (ICRA)
Dexterous in-hand manipulation (IHM) for arbitrary objects is challenging due to the rich and subtle contact process. Variable-friction manipulation is an alternative approach to dexterity, previously demonstrating robust and versatile 2D IHM capabilities with only two single-joint fingers. However, the hard-coded manipulation methods for variable friction hands are restricted to regular polygon objects and limited target poses, as well as requiring the policy to be tailored for each object. This paper proposes an end-to-end learning-based manipulation method to achieve arbitrary object manipulation for any target poses on real hardware, with minimal engineering efforts and data collection. The method features a diffusion policy-based imitation learning method with co-training from simulation and a small amount of real-world data. With the proposed framework, arbitrary objects including polygons and non-polygons can be precisely manipulated to reach arbitrary goal poses within 2 hours of training on an A100 GPU and only 1 hour of real-world data collection. The precision is higher than previous customized object-specific policies, achieving an average success rate of 71.3% with average pose error being 2.676 mm and 1.902 degree.
Inspired by the ability of human fingertips to selectively slide and grip objects, the VF-hand can achieve the same by actively switching its finger surface between high and low friction states. This allows objects to be slide or rolled along the finger surfaces.
🔗 VF-hand Link: https://www.eng.yale.edu/grablab/openhand/model_vf.html
This is trained with object-specific demonstrations collected via the same smooth-optimized exploration RL policy trained on cube. For each object, we collect 10,000 simulation demos (30 min/object).
Small Cube
Large Cube
Cube Cylinder
Three Cylinder
This is trained with a dataset of mixed demonstration of multiple objects: cube, cube cylinder, three cylinder, star, hexagon. For each object, we collect 10,000 simulation demos as before.
Seen Object
Unseen Object
Unseen Object
Unseen Object - Failure
Task: the variable-friction hand needs to continuously in-hand manipulate various objects to random target poses.
Success Criteria: Positional difference < 5mm AND Orientational difference < 5.73 degrees (0.1 radians)
Task: the variable-friction hand needs to continuously in-hand manipulate irregular objects to random target pose in air
Success Criteria: Positional difference < 5mm AND Orientational difference < 5.73 degrees (0.1 radians)
Cube Cylinder
Three Cylinder
We thank members of the MTL lab at Imperial for their support and feedback. We also thank Dr Edward Johns, Dr Sonali Parbhoo, Dr Qinghua Liu for helpful discussions.