Site still under construction

Learning Contact Dynamics through Touching: Action-conditional Graph Neural Networks for Robotic Peg Insertion

Zongyao Yi, Joachim Hertzberg and Martin Atzmueller

Preprint

Code

Datasets

Abstract: We present a learnable physics-based predictive model that provides accurate motion and force-torque prediction of the robot end effector in contact-rich manipulation. The proposed model extends the state-of-the-art GNN-based simulator (FIGNet[1][2]) with novel node and edge types, enabling action-conditional predictions for control and state estimation in the context of robotic peg insertion. Our model learns in a self-supervised manner, using only joint encoder and force-torque data while the robot is touching the environment. In simulation, the MPC agent using our model matches the performance of the same controller with the ground truth dynamics model in a challenging peg-in-hole task, while in the real-world experiment, our model achieves a 50% improvement in motion prediction accuracy and 3× increase in force-torque prediction precision over the baseline physics simulator. Furthermore, we demonstrate our model’s potential usage for state estimation by using a simple particle filter to track the robot end effector during real-world peg insertion.

Motivation

Pose estimation

efficient for sampling-based methods

State estimation

physics-plausible motion model, i.e., no object-penetration
precise observation model for F/T measurement updates

Model-based control

learnable predictive model that captures real-world contact dynamics

Pipeline overview

Action-condictioned graph

Action-conditional edges between the world nodes and the tool mesh nodes (top). Face interaction mesh-mesh edges are created when two faces fall within a predefined collision sphere (bottom).

Data collection in simulation

We build a simulation environment in MuJoCo that mirrors our real-world setup: a UR10e robot with a primitive tool mounted on its end effector. During data collection, tool shapes are randomly chosen from three primitives (triangle, square, and hexagon). To diversify the training scene, three to six box obstacles of varying dimensions are placed and fixed on the table. The robot is driven by an impedance controller

Peg insertion in simulation

We aim to assess whether the trained model enables effective model-based control in contact-rich manipulation tasks, specifically peg insertion. For this task, we use the same UR10e robot and impedance controller as during data collection, mounting one of the 4 tool shapes on the robot’s end effector: triangle, square, hexagon, or circle. A matching slot with a nominal clearance of 5mm is fixed on the table. We also examine the impact of domain-specific data by augmenting the training set with peg insertion rollouts. The fine-tuned model is labeled as Act-FIGNet-F, while the model trained on the augmented dataset is labeled as Act-FIGNet-A

Insertion success rate comparison

The MPC agent using the learned model achieve up to 98% success zero-shot. At the tightest 1 mm clearance, ﬁne-tuning on self-collected data brings it on par with an agent planning with the ground truth dynamics

Zero-shot transfer to out-of-distribution actions

Real-world setups

Real-world experiments are performed on a UR10e robot fitted with a high-precision force-torque sensor and a custom 3D-printed tool mounted on its end effector. For this experiment, we collect data with two tool geometries, i.e., hexagonal and circular, against a complementary hexagonal slot with a nominal clearance of 1mm fixed on the test table

Control pipeline

the random agent generates target wrench sent to the controller. The controller implements closed-loop force control, using the force-torque feedback and joint positions to calculate target joint positions sent to the robot. Tool pose is estimated via forward kinematics. Since data collection and the control loop run at different frequencies, a filter is applied to synchronize and smooth the recorded data, i.e., F/T readings and tool pose.

Data collection with random actions

We record the control inputs as well as the end effector pose, velocity, and F/T readings. In total, 450k steps are sampled, of which 400k for training and 50k for testing

Motion and force prediction on an exploration rollout

Zero-shot transfer to peg insertion actions

Peg pose estimation

Tracking the peg pose during peg insertion with a simple particle filter that uses the learned dynamics model for motion updates, and the learned observation model for F/T measurement updates.

Rendered images of the a real-world peg insertion episode, overlaid with the particle filter estimates. The green tools represent ground truth, while the transparent magenta overlays represent the particle estimates/motion prediction. The magenta prediction on the left is almost invisible before making contact because it closely follows the ground truth. On the center and the right, due to the concentration of particles around the ground truth, the magenta overlays appear as a single solid body occluding the green ground truth.

Socket pose estimation

Follow-up work on 6D socket/object pose estimation using wrench data alone is coming soon. Please follow our updates.

Position only

Full 6D pose estimation

Supplementary video

act-fignet-video-v5.mp4

Google Sites

Report abuse