Allegro Hand Object Manipulation

The cube manipulation task constantly randomizes the goal pose of the cube that the Allegro Hand must achieve. The model was trained in IsaacSim using Proximal Policy Optimization (PPO) reinforcement learning. I am now working on transferring the policy contained within the simulation to the real robot (Sim-to-Real).

For state observation, I created a program using an Intel RealSense camera and OpenCV detection libraries for ArUco markers. If at least one ArUco marker is clearly visible to the camera, it is able to accurately estimate the 6D pose of the cube with respect to the world frame. This cube detection program runs on a node contained in ROS with the results being visualized in RViz.

The motors of the Allegro Hand are controlled through a CAN bus device, with each motor being addressable through a CAN message stream. I am now working on integrating all of the functional components to work together. I need to have the cube detection program publish messages through ROS to the policy which receives observations and responds by publishing the desired joint positions (actions).