5 Robots Named Paul

Human studies

The robots I develop are influenced by research into human behaviour, more specifically how human beings depict other humans, how humans perceive artworks and how humans relate to robots. The artifacts produced by these computational systems can also be considered as studies of the human.




5 Pauls drawing Patrick
5 robots named Paul sketching Patrick (Paul-III.(a,b,c,d,e)), 2012, photo by Tommo



5 Robots Named Paul

In a scene reminiscent of a life drawing class, 5 robots named Paul await the human sitter. When the subject arrives he is invited to sit in an armchair, an assistant pins sheets of paper onto the robots and wakes each one up. Immediately the robots look at the sitter and start to draw, their gazes alternating between the drawing in progress and the posing human. As the model in a life drawing class, the sitter is an object of study. Immobile, yet active in keeping the pose, the human is there to inspire the machines. For the audience he is only one of the 6 silent actors of a short theatrical event. The sounds produced by the robot’s motors create an improvised soundtrack. The robots, stylised minimal obsessive artists, are only capable of drawing. Each look alike except for their eyes, either obsolete digital cameras, or webcams. Their bodies are old school desks on which the drawing paper is pinned. Their left arms, bolted on the desks and holding black biros, are only able to draw. During the exhibition day after day the drawings progressively cover the gallery’s walls. Paul’s behaviours are based on research into the cognitive, perceptual and motor processes involved when artists draw from life, and also by the author’s drawing practice. The software driving Paul written, by Tresset is implemented using research from robotics, computer vision and computational creativity. Even if the way Paul draws is based on Tresset's own technique, its style is not a pastiche but rather an interpretation influenced by the robot’s characteristics.


See this paper for more information on Patrick's research work: P. Tresset, O. Deussen,  "Artistically Skilled Robots", proceedings of the AISB50 conference, 2014. PDF

The 5 Robots Named Paul installation was created for the Merge Festival 2012, London



album




Paul


Paul is an obsessive artificial drawing entity that sketches people who pose for it. Its sketching style resembles Patrick Tresset's own. As a robotic installation it can be considered from a number of perspectives:
At a superficial level Paul is akin to the court automatas of the 18th century and to 19th century fairground attractions, attracting fascinated adult crowds and amazed children. Each sitter poses for Paul for a 30 minute session and the robot plays with the sitters attention theatrically, alternatively shifting its attention from the drawing in progress to the subject's face.
As Paul's ancestors were initially developed by Patrick to palliate the loss of creative ability, the robotic system can be seen as a form of artistic prosthetic. Paul's drawing style and behaviour represents a portion of Patrick's identity and can therefore be seen as an existential self portrait.
Paul produces drawings that are considered as artworks it has to date drawn more than 3000 individuals have experienced been sketched by a Paul. One of its sketches is in London's Victoria and Albert Museum's collections. As a generative system it can be considered under the same light as the pioneering work of algorists such as Roman Verostko and Harold Cohen. As with all generative work, the system raises questions of authorship.
Paul is also a research tool, the robotic system and some of the computational elements of the system were developed in the context of AIkon-II, a sci-art research project investigating the perceptual, motor and cognitive processes involved in observational drawing.
Paul is developed using sophisticated technologies from research fields such as computer vision, artificial intelligence, cognitive computing and robotics
From a speculative perspective, it is predicted that in the foreseeable future we will have companion robots in our homes. As we are still some way from artificial consciousness these robots will encounter problems of social integration, it is likely that artistic or entertaining skills will help them to fit in.


Paul's physical description


Paul is composed of a left-handed robotic arm holding a black Biro pen and an actuated pan and tilt web-cam (eye) bolted to a table. On one side of the table is a chair. Always present with the installation is an assistant: his/her role is to change the paper and give the signal to Paul that somebody is ready to be sketched which is achieved by covering the camera for ten seconds. The assistant may also give directions to the sitter and adjusts the light. Generally, when space permits, unsold sketches are displayed on the wall around or behind the installation. Paul also performs a number of behaviours that are only pretences. These actions are implemented to make the audience believe that Paul is more alive than it is, and reinforce the relation between the sitter and Paul. For example whilst sketching, Paul often takes a look at the sitter, scanning the face with multiple saccades and fixations. Paul’s eye also follows the tip of pen’s movements during the drawing phases. 





The drawings

From the onset the strategy has been to implement a series of processes, mimicking what I would have done by hand, rather than trying to get a specific graphical result. Using this strategy is the best way to obtain a rich coherent style. This manner of approaching the problem is in accord with the idea that artworks are memories of the processes that created them. It is thought that a significant part of our emotional response to an artwork when looking at it is caused by our capacity to overtly and covertly recover the artist's actions and intentions [11, 15, 13, 14]. As Michael Leyton argues, artworks are maximal memory stores [8]. Particularly, at the low level it seems that our neural mirror system allows us to mentally re-enact the gestures that have contributed to an artwork's creation [3, 4].
Drawing perception, the manner with which a drawing operates on an observer is complex with the ensemble of lines traced on paper operating at different levels on the observer. Perceiving a drawing is an exceptional and unnatural visual experience. When we observe a natural scene, multiple areas of the visual cortex are activated, reacting to colour, movements and so on. When we observe a drawing, no colour or movements can be found, only the line elements which are exceptionally found in the natural environment. In the early visual cortex, the lowest level of the perceptual system, cells react strongly to lines/edges with certain orientations and strong contrast [5]. Furthermore, a stimulus activating only one region of the visual cortex is stronger than if activating multiple regions. 
A number of factors can account for the perceived quality of Paul's sketches, such as the choice of paper, layout and composition. When Paul draws lines, their paths are extracted from the responses of Gabor filters [12]. Such filters are known to be good models of simple cells in the early visual cortex (V1)[7] and as a consequence accentuate what would be perceived as salient features [6]. The use of simulated visual feedback to constrain and evaluate the random exploration during the shading process is sufficient to produce patterns that are perceived as being the result of an intentional process. Paul's drawings are the result of a sequence of movements and as such they are the record of a process. Evidence that the traces forming part of a drawing by Paul are the results of movements can be found in the lines irregularities. Although these irregularities are not akin to the imperfections a human might produce, they have characteristics that could only been the result of a pen in motion driven by an articulated arm. Furthermore, the layering of successive lines and of successive shading patterns adds to the drawing being perceived as the result or consequence of a sequence of intentional movements/processes.


Related Press Coverage


Le Monde, France, Catherine Mary, December 1,  2012:


Time Out, United Kingdom, Sonia Barber, October 16, 2012


The Times, United Kingdom, Daisy Greenwell, October 11, 2012


Slate, USA, Torie Bosh, Novembe 15, 2012


Related Academic Publications

P. Tresset, O. Deussen,  "Artistically Skilled Robots", proceedings of the AISB50 conference, 2014

P. Tresset, F. F. Leymarie, “Portrait Drawing by Paul the Robot”, Computers and Graphics, volume 37, 2013.

P. Tresset,  F. F. Leymarie,”Human Robot Interaction and Drawing”,Workshop on “Robot Feedback in Human-Robot Interaction”, part of RoMan, the 21st IEEE International Symposium on Robot and Human Interactive Communication, Paris, France, September 2012.

P. Tresset, F. F. Leymarie, “Sketches by Paul the Robot”, Proceedings of the International Symposium on Computational Aesthetics in Graphics, Visualization, and Imaging, D. Cunningham and D. House, ed.., pp. 17-24, Annecy, France,  June. 2012, (Best paper award).

P. Tresset, F. F. Leymarie, “Paul the Robot as a Naive Drawer”, Thinking through drawing: Practice into knowledge, A. Kantrowitz, A. Brew and M. Fava, ed., pp. 103-108, New York, NY, USA,  Oct. 2011.

P. Tresset, F. F. Leymarie and N. Khaorapapong, “Skediomata: Guinea Pig and Performer”, Proceedings of ISEA 2011, Istanbul, Turkey, Sept. 2011.

P. Tresset and F. F. Leymarie, “AIKON: the Artistic/automatic IKONograph”, ACM SIGGRAPH 2006 Research posters, SIGGRAPH’06, Boston, MA, USA, 2006.

P. Tresset and F. F. Leymarie, “Generative Portrait Sketching”,  Proceedings of the 11th International Conference on Virtual Systems and Multimedia (VSMM), H. Thwaites, ed., pp. 739-748, Ghent, Belgium, October 2005


Software

Constantly evolving (first version 2009).

See this paper for details:  P. Tresset, F. F. Leymarie, “Portrait Drawing by Paul the Robot”, Computers and Graphics, volume 37, 2013. PDF



The software driving Paul is based on technologies developed by Tresset in the context of AIkon-II, a doctoral research project hosted within Goldsmiths College’s computing department. The AIkon-II project investigated the observational sketching activity through computational modeling and robotics. The software is based on research findings from fields such as robotics, computer vision, AI, cognitive computing and computational creativity. The system is not programmed to get specific results but rather an ensemble of processes are implemented, their successions and interactions eventually producing drawings based on what the system is perceiving. The software is implemented on Linux and programmed in Python and C++ , its architecture is based on the YARP framework, a platform originally created for the i-cub, and it uses specialised open source libraries such as OpenCV, Gamera, Skilearn, Scipy (see bellow for details). The software, now very stable, is modular which allows for easy extension, it also allows for multiple robots or computers to seamlessly communicate over networks, for example to add computational power or to synchronise behaviours. Each time a Paul is exhibited the software is modified to be exhibition-specific. This adds another level to the installation making each series of drawings linked to and influenced by the location, the people and Tresset’s mood.


Paul's software uses:

OS: Linux Ubuntu 10.04
Main languages: python, c++
Robotic middle-ware: YARP
Scientific library: Scipy
Machine learning: Scikit learn
Computer vision: OpenCV
Image analysis: Gamera toolkit
Geometry: Shapely

Paul's software developer: Patrick Tresset



5 Robots Named Paul first exhibition was supported by






References


[1] Harold Cohen. How to draw three people in a botanical garden. In Proceedings of the Conference of the Association for the Advancement of Artificial Intelligence (AAAI), pages 846855, 1988.
[2] Oliver Deussen, Thomas Lindemeier, Sören Pirk, and Mark Tautzenberger. Feedback-guided stroke placement for a painting machine. InProceedings of the 8th International Symposium on Computational Aesthetics in Graphics, Visualization, and Imaging (CAe), pages 2533, Annecy, France, June 2012. 
[3] David Freedberg and Vittorio Gallese. Motion, emotion and empathy in esthetic experience. Trends in Cognitive Sciences, 11(5):197203, May 2007.
[4] Victorio Gallese and David Freedberg. Mirror and Canonical Neurons are Crucial Elements in Aesthetic Response Mirror. Trends in Cognitive Sciences,11(10):20067, 2007.
[5] David H. Hubel and Torsten N. Wiesel. Receptive fields of single neurons in cat's striate cortex. In the Journal of Physiology, volume 148, pages 574591. Physiological society, London 1959.
[6] Laurent Itti, Cristof Koch, and Ernst Niebur. A model of saliency-based visual attention for rapid scene analysis.IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(11):12541259, Nov 1998.
[7] John P. Jones and Lawrence A. Palmer. An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. Journal of Neuro-physiology, 58(6):12331258, December 1987.
[8] Michael Leyton. The Structure of Paintings. Springer, 2006.
[9] Yan Lu, Josh Lam, and Yeung Yam. Preliminary study on vision-based pen-and-ink drawing by a robotic manipulator. In Proceedings of the IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pages 578583, Singapore, 2009.
[10] Pamela McCorduck. AARON'S CODE: Meta-Art, Artificial Intelligence and the Work of Harold Cohen. W. H. Freeman, 1990.
[11] Deanna Petherbridge.The Primacy of Drawing: Histories and Theories of Practice. Yale University Press, 2010.
[12] Nikolay Petkov. Biologically motivated computationally intensive approaches to image pattern recognition.Future Generation Computer Systems, 11:451465, 1995.
[13] Alessandro Pignocchi. Perceiving the Intentional Properties of Pictures. 2007.
[14] Alessandro Pignocchi. How the intentions of the draftsman shape perception of a drawing. Consciousness and Cognition, 20:887898, 2010.
[15] Philip Rawson. Drawing. University of Pennsylvania Press, USA, 1987.
[16] Patrick Tresset and Frederic Fol Leymarie. Generative portrait sketching. Proceedings of VSMM, 11th International Conference on Virtual Systems and MultiMedia, pages 739748, Ghent, Belgium, 2005. Archaeolingua publishers.
[17] Patrick Tresset and Frederic Fol Leymarie. AIKON: The Artistic/Automatic IKONograph. In Proceedings of the ACM International Conference SIGGRAPH, Boston, USA, 2006. Research poster.
[18] Patrick Tresset and Frederic Fol Leymarie. Portrait drawing by paul the robot. Computers & Graphics, 2013. Elsevier publishers.