The ROS packages that we had used for this project was the moveit_commander to set up constraints for the joints on the sawyer right arm as well as the intera_interface to load the model of the sawyer and the images that were captured by the sawyer camera.

As shown in the diagram above, we included:

• a listener
• a node
• a publisher (sawyer right hand camera)
• external computer vision library

Program Flow:

1) Move the Sawyer Arm to the initial position so that its right hand camera can view the whiteboard as it looks down upon it

• Shown below is the code to constrain the sawyer bot as well as how it translated to the initial position of the sawyer right arm

2) Identify the whiteboard and the largest unfilled shape inside. There are various steps to find and identify the lines of the largest unfilled shape:

• Points are given from the right-hand camera identifying the shape
• Add more points by creating a function to find the distance between any two points and if they were more than the given distance, add points that were in the middle of the two given points and call the function recursively until all the points are a certain distance apart (image shown below)
• Create a line of symmetry for the given convex shape using the OpenCV package's fit line function, get the slope and y-intercept of the line and identify points as either on the top of the shape or the bottom as shown in the image below.

3) The system returns the image of the shape with all the pixel coordinates shown as well as the line of symmetry which divides the points into two sections. Below are the images of the largest unfilled identified shape of a smiling face we had wanted to color! In these images, the whiteboard is also identified by a line drawn on the edge of the whiteboard. We were able to view these images using OpenCV's imshow() function.

4) Transfer all these pixel points from the listener python file to the node and use a homogenous transform to transform pixel coordinates to those used by the sawyer bot to identify points on the whiteboard. Below is the method we used to create the homogenous matrix:

In the images shown above, the x and y coordinates refer to set measured coordinates of one corner of the whiteboard as shown relative to the base frame of the sawyer. The u and v points refer to the pixel coordinates of the corner of the whiteboard.

Each of the x and y points refer to all the four corners of the whiteboard: top left, bottom left, top right, and bottom right. Each of the u and v points refer to the same corners of the white board but in pixel coordinates.

Use: Ax = b matrix formula

x and y points were measured using tf-echo relative to the base frame while the u and v points were measured as the pixel coordinates for each of the corners of the whiteboard. This way, we created the 'A' matrix and 'b' vector and found each of the elements by multiplying the inverse matrix of A by the b vector.

A^(-1) * b = x

Plug in each of the values found using that multiplication into the homogenous matrix shown above to create the matrix. However, this maps base frame coordinates into image pixel coordinates.

To fix the mapping to be from the image pixel coordinates into the base frame coordinates, get the inverse of the homogenous matrix. Then, multiply each of the pixel coordinates by the resulting "Q" matrix (above).

Q * (top pixel coordinates)

Q * (bottom pixel coordinates)

5) Draw lines from the top coordinate points to the bottom points. The points are shown in a counter-clockwise order of the shape, so we traversed the top coordinates in order of the list and bottom coordinates in backwards order in the list. This is shown in our python file below: