Design & Methods

The audio from the Locobot's microphone is passed through Google's Speech-Text API to acquire the target object's name. This is passed through a YOLO-V11 object detection model along with the RealSense camera's real-time RGB-D image stream. If the object is in the camera frame, the planner publishes the center coordinates of the object.

The target pose obtained from the object detection center coordinates is used to determine whether the robot's position and orientation is below the gripper reachable thresholds. If so, the robot is close enough to grasp the object, and publishes the flag directing the arm to begin gripping, but if not, it uses a P controller to move closer to the target object.

With the center coordinates obtained from object detection, the gripper can move to a hover height above the object, open its jaws, move to the target pose on the object, close its jaws tightly around the object, and lift the object to its predesignated lift height.