The artworks below show initial attempts at pointillism artworks. From the results, it is apparent that while the dithering algorithm shows promise, the information deposit rate of this technique is not sufficiently high. In other words, whilst a recognizable image is possible with a high enough number of points, it is not a practical approach.
The first drawing attempted using a dry erase marker on whiteboard. The marker dried out before the artwork could be completed.
A shaded sphere represented using a pointillism technique. The shaded sphere is a good test subject as it features both sharp edges (resolution controlled) and gradients (layering controlled).
In order to increase the information deposit rate, the team transitioned to a cross hatching algorithm, for which a selection of results is shown below. Note in the early iterations of the sphere that there is an overshooting problem at the edges of the subject due to the use of open-loop position control over closed-loop control such as PID. The overshooting errors were mitigated by reducing the joint speeds, however, future work should implement a more sophisticated dynamic control method that reduces drawing errors.
Three iterations of the shaded sphere are shown above, with an increasing number of lines and layers. Using the same test subject as in the dithering tests allow for direct comparisons, and for the evaluation of strengths and weaknesses of each method.
We proceed to present the results for using a human face as the subject, a key objective/priority of the project. Faces are challenging to draw as there is subtlety in the shading, such as between the chin and neck, as well as small details such as eyes and nose. However, the difficulty is in part reduced because faces are one of the subjects humans are the best at recognizing and judging, and we are used to interpreting abstractions of faces from the large variety of animated media consumed. Showing the image to acquaintances of the subject is therefore key in assessing the quality of the drawing.
We note the interesting observation that the perceived quality of the image seems to increase as the image is scaled down/viewed from further away, as the inaccuracies and texture of the drawing disappear, making the portrait more organic.
A reproduction of The Son of Man is shown below. The image is a good test subject as it combines a human subject with clear colors and contrast. The resulting image shows that the color mixing is imperfect, whilst also showing promise. In particular, the color of the green apple is not reproduced accurately, though some green can be seen in the image, indicating that it is indeed possible to reproduce. Furthermore, the compromise in the number of lines per layer and the number of layers per color is apparent in the reduced clarity of the details. The compromise is necessary as the four colors require at least four times as many layers.
Finally, we present a reproduction of The Starry Night, which is by far the most challenging subject due to the combination of colors, detail, and texture.
Overall the results of our project turned out better than expected, especially with the cross hatching drawings. We were able to recreate a variety of different styles of images and have the subject of the images be recognizable. We were also able to go beyond our initial goals of drawing with pointillism and managed to achieve fully automated multicolor cross hatching.
There were a significant number of challenges and difficulties that we faced that, and we weren't able to come up with ideal solutions for all of them.
The control aspect of the Sawyer arm did not have as much precision as expected, and this has led to a number of issues. We found that we could only manipulate joint angles to be within 0.005 radians of the desired angle, but this could potentially produce errors at least half a centimeter away from the desired position. In order to mitigate this, we tried decreasing the velocity of the joints, but this didn't appear to help. However, due to the nature of cross hatching not producing perfectly detailed and accurate drawings, this noise appeared acceptable in the resulting drawings.
The length of time it takes to draw pictures is quite a bit longer than expected. This issue was more pronounced when we were initially doing pointillism, but it ended up being somewhat mitigated through cross hatching. We tried adjusting various parameters to speed up the drawing, such as rearranging the order of the cross hatching lines being drawn, which appeared to be quite beneficial, and speeding up the joint velocity, which caused a degrade in the quality of the drawing and wasn't ideal. Because of the inertia of the arm itself, the pen would overshoot the end of the line at the higher joint speeds and draw longer lines than intended. One improvement that we did find was to apply a Gaussian blur before generating the cross hatching lines, which reduced the number of lines and also mostly preserved the quality of the resulting image.
Drawing straight lines was always a challenge especially with our control aspect only dictating the position of the end effector. We tried to draw lines as straight as possible by adding multiple waypoints along the line for the end effector to reach in order to correct for deviations, but due to the lack of precision with Sawyer, it wasn't able to straighten the lines much. With too many waypoints (e.g. one every mm or two), we found the lines to be very jagged, so we opted to go for fewer waypoints to create a more artistic feel with smoother but wavy lines. Again, this noise in the lines appeared acceptable in the resulting drawings.
Height calibration was the only manual process we had to do whenever we wanted to draw anything. Ideally, this was something we could automate into our code along with the canvas localization, but this remained an unresolved issue throughout our project. We were unable to use AR tags to determine the height of the canvas due to the substantial error margin along with the variety of different length pens. One option we didn't explore were integrating some pressure sensor into the end effector, but this was infeasible to incorporate into our end effector design and have connected to the rest of our setup. Another option was to use an external camera to visibly detect whether the tip of the pen touched the canvas, but this computer vision approach would require significant engineering effort beyond our primary project scope.
Although most of our resulting drawings as shown above resembled the source images quite closely, there were limitations on which kinds of images we could draw. The main limitation was that small shapes and details could not be accurately captured in our line drawings, since the primary purpose of cross hatching is to create the proper color or shading of a region. In order to remedy this, we tried determining edges of the image through computing the gradient or using Canny edge detection, but the effects of these edge lines appeared to only give minimal improvement in the quality of the resulting image compared to the number of additional lines drawn, which would significantly increase the time taken to draw the image.
For the multi-color end effector, there were significant delays in the manufacturing process due to long queues for the 3d printer. As a result, some parts of the end-effector were not printed. In particular, the revised versions of the pen clamps with additional clearance were not completed, and pen clamps from the first iteration had to be modified (manually) to fit the end-effector. As a result of this change, other parts also had to be modified, which included manual sanding/filing of rod clamps, drilling and punching of wooden offsets, and more. The modifications were successful and the design is fully functional, though the performance of the end-effector would have been better with more traditional manufacturing techniques such as milling and turning.
We chose to use the subtractive colors cyan, magenta, yellow, and black (CMYK) because of the behavior of light when adding pen ink. However, our sketch of The Starry Night turned out very dark.
We hope the artworks will inspire creative souls to enter the field of robotics, and that the public perception of the area is changed for the better.