Allegro Hand Object Manipulation

The cube manipulation task constantly randomizes the goal pose of the cube that the Allegro Hand must achieve. The model was trained in IsaacSim using Proximal Policy Optimization (PPO) reinforcement learning. I am now working on transferring the policy contained within the simulation to the real robot (Sim-to-Real).

For providing observations of the cube's orientation during policy inference, I created a ROS2 node that uses the Intel RealSense camera and OpenCV for detecting ArUco markers. If at least one ArUco marker is visible to the camera, it is able to accurately estimate the 6D pose of the cube with respect to the hand's base frame. The results of this cube state estimator were visualized in RViz.

The motors of the Allegro Hand are controlled via CAN bus. I am still working on completing the control loop of applying the policy actions to the motors.