In order to determine where the canvas is in the robot's coordinate frame, we attached 4 AR tags to the corners of the canvas region. We used the camera on Sawyer's right hand to have it scan a predefined region for AR tags and used the ar_track_alvar package in order to detect the transformations of all AR tags with respect to the camera. We found that this scanning procedure was necessary due to the fact that placing the camera too high to capture all 4 AR tags resulted in them being barely visible and undetectable from the low quality camera. Combining this with the known transformation from the camera to the base of Sawyer gave the coordinates of each AR tag with respect to Sawyer's coordinate frame.
We determined the center of the canvas to be the average of the coordinates of all 4 AR tags, the width of the canvas to be the difference between the x-coordinates of the middle 2 AR tags along the x-direction, and the height of the canvas to be the difference between the y-coordinates of the middle 2 AR tags along the y-direction. Although we could also obtain the z-coordinates and thus the height of the canvas being drawn at, we found that the error margin of the detected z-coordinate was too high to allow for reliable drawings, and we often changed the pen and pen length we drew with, so we resorted to manual height calibration of the pen instead of using AR tags to automate that.