Embedded Files
Joint Detection - YOLO
In order to determine the movement of the person 'dancing,' we used an OpenCV model called YOLOv8, developed by Ultralytics. YOLOv8 is able to detect a person's joints, referred to as keypoints. We selected the nose, left/right wrist, and left/right ankle to be the five keypoints to track. At a high level, we feed the video (webcam or prerecorded video) into our model, and it outputs an array of keypoints for every frame. Conveniently, it can identify each keypoint and keep them consistent over time (for example, it recognizes the left shoulder in each frame and labels it accordingly). Every time YOLOv8 processes a frame, we extract the five tracked keypoints, and the data is formatted for publication to the movement node.
Controlling the Neatos
Neatos are vacuuming robots, serving as a convenient platform for testing code when controlled by a Raspberry Pi. Within our fleet of five Neatos, each robot represents a keypoint, including the head, arms, and legs. To control each Neato and provide separate movement instructions, we created a movement publisher node correlated with a specific Neato. Additionally, our movement node had a subscriber linked to each Neato to obtain odometry information. To efficiently store Neato-specific information, we used a dictionary containing specific details for each Neato. When the movement node received the latest set of keypoints, it assigned the delta between the current position of the Neato, given by the odometry, and the position of the target keypoint. It then published the corresponding commands to each Neato to turn to the correct heading and move the correct distance.
Initializing Neatos and Creating a Global Map
The webcam captures a grid of 650x500 pixels, which we decided to represent as a 3.25 x 2.5-meter frame for the Neatos. Every keypoint is converted to the map frame, and the received odometry of each Neato is also converted to the map frame. The code assumes that each Neato starts at the correct position and orientation. Upon receiving the first set of keypoints from the keypoint processing node, the location of each Neato on the global map is set to the corresponding tracked keypoint. As each Neato's location is tracked via odometry, the transformation from the global map to odometry is recorded for each Neato.
Page updated
Google Sites
Report abuse