Embedded Files
This project is being developed in collaboration with Luis Alvarez, Universidad de Las Palmas Gran Canaria. Have also contributed Bruno Galerne, Yann Gousseau and Agustin Salgado de la Nuez. Its goal is to develop the mathematical and computation theory of abstract images, and to apply it to the development of a generative theory of decorative art.
You can try on line our abstract shape and texture generators and see many experimental results at jmandla.com
Before explaining the project, I'll show several images obtained without drawing, by a computer program making random choices and generating automatically the image. The only input of the program is a small set of silhouettes (cat, swan,...) and a color image that serves as color palette. This means that when I show a series of images, it has been successively and randomly generated. There is, however, another human intervention: the images have been sieved out, obviously by esthetic criteria that are personal. I therefore indicate as a percentage the approximate proportion of images that have been selected in the series produced by the computer.
Figure 1: Variations, painting in a cat silhouette, random texture, black and white palette
In Figure 1 I show a cat silhouette with random textures painted into it; the texture adapts to the containing shape, thus following the gestalt law that the "whole conditions the parts". Figure 2 is another illustration of the same painting principle, applied to a swan silhouette and with varying color palette.
Figure 2: Variations, painting in a swan silhouette, random textures, three different color palettes
The color palette is an input image used to select the color in the random generation. Figure 3 shows a color palette followed by random images respecting this palette.
Figure 3: abstract variations on a Van Gogh palette. Left, the palette, right, two random images.
A last preliminary illustration (Figure 4) shows the influence of the (random) choice of the painting technique on the final result. Starting from simple shapes and a palette, the random images can combine abstract painting techniques with exclusion, inclusion, transparency, tesselation, perspective, among other effects.
Figure 4: abstract variations with a palette also chosen randomly. The second palette is black and white. The third synthesis shows random Mandalas obtained with a variant of our image synthesis software, also used for designing scarfs (see Figure 12). The fourth image uses exclusively a random tessellation technique and the last one a transparency technique.
Perspectives on image sciences
Figure 5: Exemplar based texture sampling. Left: Portilla-Simoncelli’s method applied to a textile image. Right : Galerne et al. method applied to sea water and gravel on abeach.
The science of images is by nature interdisciplinary. It covers endeavours and disciplines as diverse as abstract art, decorative art, computer graphics, geometric measure theory, stochastic geometry, image processing, computer vision, gestalt psychology, and psychophysics. Using very diverse terminologies and techniques, all of these disciplines address the perceptual content of images and the ways to create or analyze them. The challenge of our project is to link all of these aspects in a mathematical and computational theory. The envisaged tools are mathematical models linked to algorithms and software permitting to explore and expand by random synthesis the set of abstract images.
This page argues that all of the above mentioned image formation and image interpretation theories rely on on few generative principles that can be geometrically formalized and simulated by random simulation. This means that one can design and visualize very general classes of images by random sampling. Thus the project aims at producing:
-an extension of geometric measure theory describing the structure of planar sets and func- tions;
-a general synthesis theory for (perceptually discriminable) artificial and natural textures;
-new sorts of abstract images expanding the technical possibilities of abstract digital art;
-a generative theory of decorative art.
Figure 6: Left (in grey): three images in Sobolev spaces images with a Fourier coefficient decay implying that they belong respectively to the Hilbert and Sobolev spaces L2, H1/2 and H1. Right: random emulation of a wood image by the Heeger Bergen model. As for Besov spaces it is characterized by a first order statisticsi of its wavelet coefficients.
Function spaces and texture synthesis
Image processing and computer graphics have revealed wide gaps in the mathematical classification of functions based on their geometric properties, even for 2D functions. Our mathematical function spaces have been designed to express variational principles modeling solids and liquids. They are associated with an energy given by some space integral. Its boundedness characterizes the elements of the function space, and it is a really rough characterization.To realize this by using computer graphics, one can generate random elements of each function space and see how they look. For example the spot noise algorithm can synthesize a wide class of micro-textures including random elements of any Sobolev space. Fig. 5 shows on the left three generic elements of three Sobolev spaces. On the right, the same algorithm shows two natural textures that can be also modeled by fixing much more precisely the decay of their Fourier coefficients moduli: waves and gravel. Similarly, texture synthesis models based on histograms of wavelet coefficients incidentally permit to simulate all elements of Besov spaces. They manage describing certain more complex textures as illustrated on the right of Figure 6 with a wood simulation. In short, some observed textures in nature are modeled by a variant of Sobolev or Besov space where the profile of the Fourier or wavelet coefficients is fixed. But how many such different textures are possible? This question makes no sense unless we give a discrimination criterion. It is an easy psychophysical experiment to check that all textures generated by spot noise with identical Fourier coefficient moduli are visually undiscriminable. (This experiment can be performed online on any image with the IPOL paper (25).)
There is a still more general version of the dead leaves model involving transparency (42). Random tessellations are another way to simulate planar texturesby simple subdivision processes (43) See Figure 7, which also shows a fractal generated by iterated function system (44).
These mathematical attempts are each focused on a single generation technique. The question of generating abstract images and drawings and exploring their perception was investigated almost simultaneously and in strikingly similar terms by abstract painters and Gestalt psychologists at the beginning of the past century. Gestalt theory (45) postulates that our perception proceeds by grouping the local percepts present in an image into more global entities, the Gestalts. The Gestalt grouping laws (46) are simple, geometric, and can be simulated numerically: proximity, symmetry, same color, sameshape, same orientation, good continuation, periodicity. Strikingly, these grouping laws are the implicit generative principles of decorative art (Figure 13),which also proceeds by combining simple shapes recursively. Kanizsa’s books actually focus on human perception of the two main painting techniques,occlusion and transparency (47).
According to Bela Julesz, two textures are called indiscriminable if they cannot be discriminated from each other by a short examination, such as two pieces of the same textile. For example all white noise images with a same mean and variance are indiscriminable, and are considered “the same”texture (26). This psychophysical theory has become computational thanks to computer graphics. Portilla and Simoncelli have invented a texture synthesisalgorithm (27) convincingly achieving Julesz’s program: it shows that a wide range of examples of natural and artificial textures are perceptuallycharacterized (and can be approximately synthesized) by about 700 statistical wavelet moments and wavelet coefficient correlations (Figure 5, left). Two textures characterized by the same parameters are indeed indiscriminable.
Figure 7: Basic mathematical processes generating abstract textures: a scaling dead leaves model a random tessellation, a fractal (iterated function system).
Structure theorems for 2D images and sets?
We have examined briefly functional spaces as image models and noticed that they could be viewed by image synthesis techniques and would boil down to rather specific texture models. A generic Sobolev function indeed looks like a cloudy sky with no geometric structure (Figure 6). But, in most images of the world, we do see a lot of geometric content!
Yet, little mathematical effort has been dedicated to the geometric structure of 2D shapes and images in general. Only two complete mathematical theories exist: Besicovitch (28) characterized the geometric structure of subsets of the plane with finite length. This theory was extended to arbitrary dimensions byMarstrand and Mattila (29). This theory does not yield information on the geometry of “visible” measurable sets, with codimension 0. Ironically, the interesting sets in Besicovitch’s theory are invisible, having zero measure projections. The visible sets of the plane have positive Lebesgue measure. They can be approximated by closed or open sets with arbitrary precision. Thus, for perceptual purposes, we could limit ourselves to classifying the geometric structures of closed and of open sets.
Mathematical morphology is probably the first attempt to do so theoretically and computationally. The analysis of photographs of geological samples led Matheron and his school30 to invent operators analyzing closed and open sets and characterizing their “granulometry”. These operators are extendable to images, which can be described up to a contrast change by the stack of their upper level sets. In 2D, one can also describe the topographic map and a Morse theory for functions with minimal regularity (31). The morphological approach is more general, but a close parent of the theory of functions with bounded variation (32). Being the only theory accounting for a geometric structure in images, this was used very early in image processing (33). The level sets of a BV image have finite perimeter and have been thoroughly described (34). Yet, this does not account for the natural existence of open sets with intrinsically infinite perimeter, like irrigation networks. Such irrigating sets or networks, which are essential in anatomy and geology, are dense connected open sets with arbitrarily small measure. They arise as solutions of optimal transport problems (35).
(24) Heeger, D. J., and J. R. Bergen. “Pyramid-based texture analysis/synthesis.”, Proc. ACM, 1995.
(25) Galerne, B., Gousseau, Y., and Morel, J. M. (2011). Microtexture synthesis by phase randomization. Image Processing on Line.
(26) Julesz, B. (1981). Textons, the elements of texture perception, and their interactions. Nature, 290(5802).
(27) Portilla, J., and Simoncelli, E. P. (2000). A parametric texture model based on joint statistics of complex wavelet
coefficients. IJCV, 40(1), 49-70.
(28) Besicovitch, A. S. “On the fundamental geometrical properties of linearly measurable plane sets of points (III).”
Mathematische Annalen 116.1 (1939): 349-357.
(29) Mattila, P. (1999). Geometry of sets and measures in Euclidean spaces: fractals and rectifiability (No. 44). CUP.
(30) Serra, J. (1982). Image analysis and mathematical morphology, v. 1. Academic press.
(31) Caselles, V., and Monasse, P. (2010). Geometric description of images as topographic maps, Springer.
This fast description of the state of affairs in the geometry of 2D functions points out the necessity of a complete classification of closed and open sets, andof images modeled as upper semi-continuous functions. The example to imitate for such a theory is of course the analysis of BV functions. Yet, a more general structure theory for images should include a structural description of closed sets and perhaps a structure decomposition of them into Cantorian fibrillary parts and fully disconnected sets with Cantorian structure, or ”granulometries”, while non trivial open sets would be irrigation networks (36).
Figure 8: Abstract paintings by Delaunay, Kandinsky, Malevitch, Van Doesburg, Mondrian. All combine simple shapes by occlusion, exclusion, transparency, (random) positioning and favour their perceptual grouping by common features such as orientation, color, vanishing point.
Abstract art, generative models and painting techniques
Spectacular progress has be made in exemplar based texture synthesis as illustrated in Figures 6 and 5. However, research on image structure can go far beyond the simple reproduction of natural textures. One feels the need to also explore all means of creating new shapes and textures, regardless of their plausibility in the real world. The only limitation to the creation of new pictures is that they must be understandable to the human visual system. This understandability can receive a rigorous definition using Gestalt theory, which describes which percepts (gestalts can be built, ou "grouped" by human perception). Abstract art shows that there are many more sorts of perceptually understandable images than those accessible in nature, or those actually created by current mathematical synthesis methods. The exploration of the subject by mathematicians and computer scientists has been rather limited. The mathematical attempts at creating new abstract images have generally been based on a single generation principle (figure 7), as it is thecase for fractal models (37).
The simplest way to combine shapes is by addition, resulting in the spot noise model (38) used for realistic (39) texture synthesis by simple Fourier spectrum manipulations (Figure 6). A physically more natural shape combination principle is occlusion, by which objects partially hide each other. The dead leaves model (40) is the mathematical formalization of this principle. A multi-scale version (41) of this model was introduced for modeling natural
(32) Ambrosio, L., Fusco, N., and Pallara, D. (2000). Functions of bounded variation and free discontinuity problems (Vol. 254). Oxford: Clarendon Press.
(33) Rudin, L. I., Osher, S., and Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60(1), 259-268.
(34) Ambrosio, L., Caselles, V., Masnou, S., and Morel, J. M. (2001). Connected components of sets of finite perimeter and applications to image processing. J. of the EMS,3(1), 39-92.
(35) Bernot, M., Caselles, V., and Morel, J. M. (2009). Optimal transportation networks. Springer.
(36) As conjectured by the late Vicent Caselles (private communication).
(37) Barnsley, M. F., and Demko, S. (1985). Iterated function systems and the global construction of fractals. Proceedings of the Royal Society of London, 399(1817), 243-275.
(38) Van Wijk, J. J. (1991, July). Spot noise texture synthesis for data visualization. ACM SIGGRAPH Computer Graphics (Vol. 25, No. 4, pp. 309-318).
(39) Galerne, B., Gousseau, Y., and Morel, J. M. (2011). Random phase textures: Theory and synthesis. IEEE Trans. on Image Processing, 20(1), 257-267.
(40 )Matheron, G. (1968). Modele s´equentiel de partition al´eatoire. Rapport technique, Centre de Morpholo- gie Math´ematique, Fontainebleau.
(41) Lee, A. B., Mumford, D., and Huang, J. (2001). Occlusion models for natural images: A statistical study of a scale-invariant dead leaves model. IJCV, 41(1-2), 35-59
Figure 9: Basic painting principles combining elementary shapes: Occlusion, exclusion, inclusion, tessellation, transparency and perspective. Theseprinciples are obvious in all abstract paintings where the lack of figuration gives preeminence to shape combination principles, (see fig. 8).
The first technical treatises of abstract art sketch the rules for creating abstract paintings, starting from simple non-figurative shapes and colors (48). Their shape formation and painting techniques are best illustrated by their paintings themselves, which resemble Gestalt experiments. Figure 9 summarizes the painting principles as they arise spontaneously in any painting activity, and were actually formalized in abstract art. Their application on simple basic shapes in abstract paintings is easy to detect, as illustrated in the examples of figure 8. Like in abstract painting, complex textures and images are created by applying interaction laws between basic shapes. These shape interaction laws, occlusion, exclusion, transparency, tessellation, similarity, are illustrated in figures 8 and 9. Starting later in the sixties, digital art has opened the way to computer aided design and painting and to the use of simulated randomness by the very same techniques (49) (Figure 14).
Following these studies, a general question arises: how can we create more general classes of images than those existing in nature, that would still be visually understandable? This cannot be done by sampling randomly, independently and uniformly all pixels of a given image. Even if the set of white noise image realizations does contain all possible images, the perceptually significant ones are lost in the crowd, being extremely unlikely.
(42) Galerne, B., and Gousseau, Y. (2012). The transparent dead leaves model. Advances in Applied Probability, 44(1), 1-20.
(43 )Chiu, S. N., Stoyan, D., Kendall, W. S., and Mecke, J. (2013). Stochastic geometry and its applications. John Wiley and Sons.
(44) http://matthewjamestaylor.com/blog/create-fractals-with-recursive-drawing
(45) Wertheimer, M. (1923). Untersuchungen zur Lehre von der Gestalt. II. Psychological Research, 4(1).
(46) Metzger, W. Gesetze des Sehens, 1975. Verlag Waldemar Kramer.
(47) Kanizsa, G. (1979). Organization in vision: Essays on Gestalt perception, Praeger.
(48) Klee, P. (1953). Pedagogical sketchbook. Praeger; Kandinsky, W. (2012). Concerning the spiritual in art. Courier Dover Publications.
(49) Noll, A. M. (1966). Computers and the visual arts. Design Quarterly, 64-71.
Figure 10: Random textures similar to natural textures (zoom-in on the .pdf for detail).
Figure 11: Abstract but understandable random textures (zoom-in for detail).
Thus, a main goal of this project is to reuse in computer graphics the techniques of abstract painting, digital art and gestaltism to propose a more general definition of (perceptually understandable) images than those existing and to devise numerical methods to explore this image space. We observed that image formation theories are grounded on a short list of interaction principles: occlusion, transparency, exclusion, inclusion. Some of these principles have been individually investigated through mathematical models such as the spot noise or the dead leaves model, where a single principle is used to combine random shapes spread on the plane. Nevertheless, the systematic exploration of the visual possibilities offered by their combination has never been attempted, to the best of our knowledge. Moreover, the use of computers offers possibilities that are beyond the reach of classical painting techniques, especially when combined with a multi-scale structure.
On the other hand, randomizing all interactions might lead back to chaotic images, close again to white noise. This difficulty is avoided thanks to the application of Gestalt grouping principles, as illustrated in an abstract texture sythesis methods that I have proposed with several collaborators (50). To maintain perceptual understandability of the complex textures, the basic objects combined to create the texture were coordinated by some common property (same color, orientation, shape,...) or some common interaction (occlusion, exclusion, transparency). In gestaltic terms, such a unification is called a grouping. It helps produce images with enough redundancy to be understood by human perception. Some of these textures look natural and some do not (figures 10 and 11).
In short, we want to explore more general image classes than those currently observable, and to do so efficiently we want to synthesize them. This program is not new. Ever since the Neolithic, abstract figures and textures have been drawn on potteries. Oriental tapestries also are a course in abstract design. The search for new decorative patterns is present today in all decorative arts and in the textile industry. Providing artists and designers with new principled algorithms to cover surfaces is therefore a valid goal for mathematicians. The synthetic program should therefore permit to samplerandom decorative patterns with their regularities and variability, as illustrated in Figure 13 where this random combination of periodicity, symmetry with color and shape variations is at work, and hints at a generative definition of decorative art, realizable by computer graphics. As a token of what is obtainedby combining a few symmetries with fully random images, Figure 12 shows several shawls models by our texture synthesis algorithm that were printedon silk and displayed at the Design days of Ecole Normale Supérieure de Cachan.
(50) L. Alvarez, Y. Gousseau, J.M. Morel and A. Salgado, Exploring the space of abstract textures by principles and random sampling, submitted.
Figure 12: Five scarf models obtained by simple symmetrization of random textures.
Figure 13: Three fragments illustrating the common structure of abstract painting (Kandinsky) and decorative art (Klimt, a Kurdi carpet).
Figure 14: Digital art with abstract art techniques. Georg Nees 1965-1968; Michael Noll’s 1964 “Computer composition with lines”, a random pastiche of Mondrian’s1917 “Composition with lines” (Fig. 8); John Maeda’s “Florada, a computational study”. Technique: dead leaves models.
Page updated
Google Sites
Report abuse