IMPLEMENTATION
IMPLEMENTATION
Embedded Files
Description of Full System Operation
Description of Full System Operation
Perception, Planning, and Actuation Nodes
(prior to depth implementation made after demo day)
First, the robot is tucked into its neutral starting position and the cake is placed in view of the camera. The icing syringe is filled by a user to the proper volume and attached to the end effector. Then, the launch file is run to start up the camera and all accompanying perception and planning nodes. Once the launch file is running, our CV nodes detect and publish the type of shape, shape center location, and shape corner locations (or general perimeter locations for a circle) that it sees to the appropriate topics. From those topics, the trajectory planning node determines which type of design to draw, as well as the scale of the design based on shape size and the location for the waypoints based on shape center location. At this point, running the "main.py" file in our planning package goes through the list of trajectory points, computes the associated inverse kinematics problems required to update the robot's joint states from one point to the next, and adds each point to the job list. When the job list is executed, the robot arm moves to the first position in the trajectory list, and only then begins dispensing icing by slowly decrementing the position of the top end-effector gripper. Icing continues to dispense as the robot arm traces out the waypoints of the trajectory, and stops when the arm has reached its final trajectory point.
Implementation: Sensing
Implementation: Sensing
Implementing the sensing node required researching OpenCV, carefully calibrating the camera's location with respect to the robot base, and knowing what topics to look at when troubleshooting transformations.
Computer Vision Pipeline
Computer Vision Pipeline
The sensing node of the software uses computer vision in order to identify and publish information to the subsequent nodes.
Information is taken from the camera through three subscribers to topics containing information of color in the raw image, depth in the raw image, and camera intrinsics.
Preprocessing of the raw image (/camera/camera/color/image_raw) is done before shape detection. Steps include converting the image to grayscale, Gaussian blur to reduce noise, Otsu's threshold to convert the image into black and white, Canny edge detection, dilation to fill in gaps in edge detection.
In order to do shape detection, the raw image is converted from RGB to 8-bit BGR so that OpenCV functions can be used. After preprocessing, all contours in the image are found (cv2.findContours). A loop is set up to find the shape shown in the image by filtering out any noise or irregular blobs. If the shape found is the biggest one in the image, then its perimeter is found (cv2.arcLength) and the contour is simplified into a polygon (cv2.approxPolyDP) using the Douglas-Peuker algorithm. The shape is then classified and a String containing the shape name is returned. Image moments are used to find the center of the shape. Information about the shape's full contour, shape name, center coordinates and the simplified polygon vertices are stored.
To make sure the shape detected isn't a fluke, a stability filter is applied. It looks at the last 5 frames of the detection history to see if the shape has appeared multiple times, and averages the center position. This information is then published on /annotated_image, so that RViz has visual feedback overlayed on the camera image. It outlines the shape, shows the shape name, and labels the polygon vertices and center.
The information is then transformed into the robot's base link coordinate frame, so that it can be used in trajectory planning. The RealSense camera's location is measured by hand with respect to the base of the UR7e, and the transformation is hardcoded.
Aside from publishing to /annotated image, the shape name is published on /shape, the transformed center on /shape_center, and the transformed polygon vertices (or just random points on the perimeter for circles) on /raw_shape_waypoints. The /pre_trans_coords topic was created for troubleshooting.
Sensing Hardware
Sensing Hardware
A RealSense D435i stereo camera was used. For setting up the camera, we used the camera stands provided in Cory 105, along with a custom 3D printed attachment (connected to the camera stand with a nut, and taped the camera to it) to maintain a specific orientation of the camera facing the table. We initially had a large piece of paper with markers of the robot arm's base's wheels, table legs, and the camera stand legs, but found that to not be consistent enough.
Image from /annotated_image
Image from /annotated_image
Prior to preprocessing with Otsu's threshold and dilation, shape detection only worked with images contrasted against a brightly lit background (iPad screen).
Image from /annotated_image
Image from /annotated_image
With the updated preprocessing, shape detection worked even under dimmer lab lighting.
MarkerArray from /depth_markers
MarkerArray from /depth_markers
The MarkerArray could show the depths at each point (center of the shape in green, edge of the shape in blue), along with an approximated plane that connected the points together.
Sensing Update Post-Demo Day
Sensing Update Post-Demo Day
At Demo Day, demonstrations required hardcoding the z-distance from the height of the cake to the end effector. Feedback was given to improve the code so hardcoding isn't necessary, since the RealSense camera can give information on depth.
The sensing node was updated to detect the top plane of a cake, regardless of how tall or angled it is. The subscription to /camera/camera/aligned_depth_to_color/image_raw and enabling depth alignment in the launch file enabled use of depth. In shape_detector.py, a new topic called /boundary_pixel_coords published the points in pixel coordinates (prior to transformation). In trajectory_planning.py, subscription to the new topic allowed for depth to be found at the pixel coordinates, and then converted into meters. Random points are selected inside the edges so that the depths are for the surface of the shape, rather than the table (picking points on the edge consistently detected the table instead of the surface of the shape). Converted into 3D points, a plane is fitted every 5 seconds and visualized on RViz as a MarkerArray published in /depth_markers.
Implementation: Trajectory Planning
Implementation: Trajectory Planning
The trajectory planning pipeline converts camera-based shape detections into robot-executable icing paths by transforming computer-vision outputs into structured trajectory messages. It subscribes to sensing outputs that provide the detected shape type, cake center, and raw contour waypoints, and uses this information to determine how the decoration should be generated. Based on the detected cake size and orientation, the planner computes appropriate trajectory bounds so that each pattern is properly scaled, centered, and aligned on the cake surface. So far, we have been able to get accurate frosting trajectories for circular, square, and hexagonal cake shapes. Depending on the identified shape and selected operation mode, the system either outlines the detected cake boundary or generates a new decorative pattern within it. The resulting trajectories are then published to the /trajectory_list topic, where the UR robot executes them to perform the icing task accurately and consistently.
We were able to accurately and consistently create decorative patterns such as a spiral, a heart, and a star. Creating boundary patterns was a bit more difficult, and they would often not be properly aligned with the shape. To make the spiral, heart, and star patterns, we created parameterized versions of the shapes in Desmos to create waypoints, and then inserted them into our trajectory planning logic to plan out of the path for the UR7e arm to move to next.
Detecting a circle shape produces a spiral output, a hexagon shape produces a heart output, and a square shape produces a star output.
Implementation: Actuation
Implementation: Actuation
Subscribe to sensing topic /shape_center and UR7e driver readings from controller /joint_states
Trajectories determined using information from topics
Load IKPlanner to compute IK and plan trajectories
Create job queue created by appending joint states, executed with client
Action client for robot arm following trajectory
Gripper position set using socket client (separate from job queue)
Custom gripper for dispensing icing starts dispensing after first job finishes
Refer to Additional Materials for custom end effector CAD model.
Multiple iterations of adjusting gripper speed and height to get perfect pattern
Page updated
Google Sites
Report abuse