Exploring the set of abstract images

This project is being developed in collaboration with Luis  Alvarez, Universidad de Las Palmas Gran Canaria. Have also contributed Bruno Galerne,  Yann Gousseau and Agustin Salgado de la Nuez. Its goal is to develop the mathematical and computation theory of abstract images, and to apply it to the development of a generative theory of decorative art.

You can try on line our abstract shape and texture generators and see many experimental results at  jmandla.com 

Before explaining the project, I'll show several images obtained without drawing, by a computer program making random choices and generating automatically the image. The only input of the program is a small set of silhouettes (cat, swan,...) and a color image that serves as color palette. This means that when I show a series of images, it has been successively and randomly generated. There is, however, another human intervention: the images have been sieved out, obviously by esthetic criteria that are personal. I therefore indicate as a percentage the approximate proportion of images that have been selected in the series produced by the computer.

Figure 1: Variations, painting in a cat silhouette, random texture, black and white palette

In Figure 1 I show a cat silhouette with random textures painted into it; the texture adapts to the containing shape, thus following the gestalt law that the "whole conditions the parts". Figure 2 is another illustration of the same painting principle, applied to a swan silhouette and with varying color palette.

Figure 2: Variations, painting in a swan silhouette, random textures,  three different color palettes

The color palette is an input image used to select the color in the random generation. Figure 3 shows a color palette followed by random images respecting this palette.

Figure 3: abstract variations on a Van Gogh palette. Left, the palette, right, two random images. 

A last preliminary illustration (Figure 4) shows the influence of the (random) choice of the painting technique on the final result. Starting from simple shapes and a palette, the random images can combine abstract painting techniques with exclusion, inclusion, transparency, tesselation, perspective, among other effects.

Figure 4: abstract variations with a palette also chosen randomly. The second palette is black and white. The third synthesis shows random Mandalas obtained with a variant of our image synthesis software, also used for designing scarfs (see Figure 12).  The fourth image uses exclusively a random tessellation technique and the last one a transparency technique.

Perspectives on image sciences 

  Figure 5: Exemplar based texture sampling. Left: Portilla-Simoncelli’s method applied to a textile image. Right : Galerne et al. method applied to sea water and gravel on abeach.

The science of images is by nature interdisciplinary. It covers endeavours and disciplines as diverse as abstract art, decorative art, computer graphics, geometric measure theory, stochastic geometry, image processing, computer vision, gestalt psychology, and psychophysics. Using very diverse terminologies and techniques, all of these disciplines address the perceptual content of images and the ways to create or analyze them. The challenge of our project is to link all of these aspects in a mathematical and computational theory. The envisaged tools are mathematical models linked to algorithms and software permitting to explore and expand by random synthesis the set of abstract images.

This page argues that all of the above mentioned image formation and image interpretation theories rely on on few generative principles that can be geometrically formalized and simulated by random simulation. This means that one can design and visualize very general classes of images by random sampling. Thus the project aims at producing:

-an extension of geometric measure theory describing the structure of planar sets and func- tions;

-a general synthesis theory for (perceptually discriminable) artificial and natural textures;

-new sorts of abstract images expanding the technical possibilities of abstract digital art;

-a generative theory of decorative art.

Figure 6: Left (in grey):  three images in Sobolev spaces images with a Fourier coefficient decay implying that they belong respectively  to the Hilbert and Sobolev spaces  L2,  H1/2 and H1.  Right: random emulation of a wood image by the Heeger Bergen model. As for Besov spaces it is characterized by a first order statisticsi of its wavelet coefficients.

Function spaces and texture synthesis

Image processing and computer graphics have revealed wide gaps in the mathematical classification of functions based on their geometric properties, even for 2D functions. Our mathematical function spaces have been designed to express variational principles modeling solids and liquids. They are associated with an energy given by some space integral. Its boundedness characterizes the elements of the function space, and it is a really rough characterization.To realize this by using computer graphics, one can generate random elements of each function space and see how they look. For example the spot noise algorithm can synthesize a wide class of micro-textures including random elements of any Sobolev space. Fig. 5 shows on the left three generic elements of three Sobolev spaces. On the right, the same algorithm shows two natural textures that can be also modeled by fixing much more precisely the decay of their Fourier coefficients moduli: waves and gravel. Similarly, texture synthesis models based on histograms of wavelet coefficients incidentally permit to simulate all elements of Besov spaces. They manage describing certain more complex textures as illustrated on the right of Figure 6 with a wood simulation. In short, some observed textures in nature are modeled by a variant of Sobolev or Besov space where the profile of the Fourier or wavelet coefficients is fixed. But how many such different textures are possible? This question makes no sense unless we give a discrimination criterion. It is an easy psychophysical experiment to check that all textures generated by spot noise with identical Fourier coefficient moduli are visually undiscriminable. (This experiment can be performed online on any image with the IPOL paper (25).)

 There is a still more general version of the dead leaves model involving transparency (42). Random tessellations are another way to simulate planar texturesby simple subdivision processes (43) See Figure 7, which also shows a fractal generated by iterated function system (44).

These mathematical attempts are each focused on a single generation technique. The question of generating abstract images and drawings and exploring their perception was investigated almost simultaneously and in strikingly similar terms by abstract painters and Gestalt psychologists at the beginning of the past century. Gestalt theory (45) postulates that our perception proceeds by grouping the local percepts present in an image into more global entities, the Gestalts. The Gestalt grouping laws (46) are simple, geometric, and can be simulated numerically: proximity, symmetry, same color, sameshape, same orientation, good continuation, periodicity. Strikingly, these grouping laws are the implicit generative principles of decorative art (Figure 13),which also proceeds by combining simple shapes recursively. Kanizsa’s books actually focus on human perception of the two main painting techniques,occlusion and transparency (47).

According to Bela Julesz, two textures are called indiscriminable if they cannot be discriminated from each other by a short examination, such as two pieces of the same textile. For example all white noise images with a same mean and variance are indiscriminable, and are considered “the same”texture (26). This psychophysical theory has become computational thanks to computer graphics. Portilla and Simoncelli have invented a texture synthesisalgorithm (27)  convincingly achieving Julesz’s program: it shows that a wide range of examples of natural and artificial textures are perceptuallycharacterized (and can be approximately synthesized) by about 700 statistical wavelet moments and wavelet coefficient correlations (Figure 5, left). Two textures characterized by the same parameters are indeed indiscriminable.

Figure 7: Basic mathematical processes generating abstract textures: a scaling dead leaves model a random tessellation, a fractal (iterated function system).

Structure theorems for 2D images and sets? 

We have examined briefly functional spaces as image models and noticed that they could be viewed by image synthesis techniques and would boil down to rather specific texture models. A generic Sobolev function indeed looks like a cloudy sky with no geometric structure (Figure 6). But, in most images of the world, we do see a lot of geometric content!

Yet, little mathematical effort has been dedicated to the geometric structure of 2D shapes and images in general. Only two complete mathematical theories exist: Besicovitch (28) characterized the geometric structure of subsets of the plane with finite length. This theory was extended to arbitrary dimensions byMarstrand and Mattila (29). This theory does not yield information on the geometry of “visible” measurable sets, with codimension 0. Ironically, the interesting sets in Besicovitch’s theory are invisible, having zero measure projections. The visible sets of the plane have positive Lebesgue measure. They can be approximated by closed or open sets with arbitrary precision. Thus, for perceptual purposes, we could limit ourselves to classifying the geometric structures of closed and of open sets.

Mathematical morphology is probably the first attempt to do so theoretically and computationally. The analysis of photographs of geological samples led Matheron and his school30 to invent operators analyzing closed and open sets and characterizing their “granulometry”. These operators are extendable to images, which can be described up to a contrast change by the stack of their upper level sets.  In 2D, one can also describe the topographic map and a Morse theory for functions with minimal regularity (31). The morphological approach is more general, but a close parent of the theory of functions with bounded variation (32). Being the only theory accounting for a geometric structure in images, this was used very early in image processing (33). The level sets of a BV image have finite perimeter and have been thoroughly described (34). Yet, this does not account for the natural existence of open sets with intrinsically infinite perimeter, like irrigation networks. Such irrigating sets or networks, which are essential in anatomy and geology, are dense connected open sets with arbitrarily small measure. They arise as solutions of optimal transport problems (35).

 (24) Heeger, D. J., and J. R. Bergen. “Pyramid-based texture analysis/synthesis.”, Proc. ACM, 1995.

(25) Galerne, B., Gousseau, Y., and Morel, J. M. (2011). Microtexture synthesis by phase randomization. Image Processing      on Line.

(26) Julesz, B. (1981). Textons, the elements of texture perception, and their interactions. Nature, 290(5802).

(27) Portilla, J., and Simoncelli, E. P. (2000). A parametric texture model based on joint statistics of complex wavelet 

coefficients. IJCV, 40(1), 49-70.

(28) Besicovitch, A. S. “On the fundamental geometrical properties of linearly measurable plane sets of points (III).” 

Mathematische Annalen 116.1 (1939): 349-357.

(29) Mattila, P. (1999). Geometry of sets and measures in Euclidean spaces: fractals and rectifiability (No. 44). CUP.

(30) Serra, J. (1982). Image analysis and mathematical morphology, v. 1. Academic press.

(31) Caselles, V., and Monasse, P. (2010). Geometric description of images as topographic maps, Springer.

This fast description of the state of affairs in the geometry of 2D functions points out the necessity of a complete classification of closed and open sets, andof images modeled as upper semi-continuous functions. The example to imitate for such a theory is of course the analysis of BV functions. Yet, a more general structure theory for images should include a structural description of closed sets and perhaps a structure decomposition of them into Cantorian fibrillary parts and fully disconnected sets with Cantorian structure, or ”granulometries”, while non trivial open sets would be irrigation networks (36).

Figure 10: Random textures similar to natural textures (zoom-in on the .pdf for detail).

Figure 11: Abstract but understandable random textures (zoom-in for detail).

Figure 12: Five scarf  models obtained by simple symmetrization of random textures.

Figure 13: Three fragments illustrating the common structure of abstract painting (Kandinsky) and decorative art (Klimt, a Kurdi carpet).

Figure 14: Digital art with abstract art techniques. Georg Nees 1965-1968; Michael Noll’s 1964 “Computer composition with lines”, a random pastiche of Mondrian’s1917 “Composition with lines” (Fig. 8); John Maeda’s “Florada, a computational study”. Technique: dead leaves models.