representation, distributed
 
 
A distributed representation is one in which meaning is not captured by a single symbolic unit, but rather arises from the interaction of a set of units, normally in a network of some sort.
 

Details:
 
The concept of distributed representation is a product of joint developments in the neurosciences and in connectionist work on recognition tasks (Churchland and Sejnowski 1992). Fundamentally, a distributed representation is one in which meaning is not captured by a single symbolic unit, but rather arises from the interaction of a set of units, normally in a network of some sort. In the case of the brain, the concept of ‘grandmother’ does not seem to be represented by a single ‘grandmother cell,’ but is rather distributed across a network of neurons (Churchland and Sejnowski 1992). This method of representation stands in direct opposition to the symbolic representation used by adherents of classical artificial intelligence (AI).
 
To introduce distributed representations concisely it is helpful to contrast them with the more familiar symbolic representations. Symbolic representations are the easiest for us, as language users, to understand and apply. A symbol, such as a word or a number, is an everyday occurrence for most of humankind. For instance, each word contained in this essay is a symbolic representation of a particular meaning associated with that word. These symbols are joined into propositions based on the rules of a grammar. Of course, symbols are not restricted to being words or numbers, but this is their most common application in classical AI. From computational, psychological, and neurological standpoints, there are a number of shortcomings in using purely symbolic representations in modeling human behavior. I will briefly examine three of the more important limitations of symbolic representation that serve to distinguish them from distributed representations.
 
First, symbolic representations are strongly propositional. Thus, when this method of representation is used in non-language based applications such as image processing it becomes extremely difficult to explain many psychological findings. Similarly, taste, sound, touch, and smell are very awkward to handle with symbolic representations. In contrast, distributed representations are equally well suited to all modes of perception.
 
Second, symbolic representations are "all-or-none". This means that there is no such thing as the degradation of a symbolic representation, it is either there, or it is not there; this property is referred to as brittleness. The brittleness of symbolic representations is highly psychologically unrealistic. People often exhibit behavior that indicates partial retrieval of representations, as in the "tip-of-the-brain" recall, prosopagnosia (loss of face recognition), and image completion.
 
Furthermore, minor damage to a symbolic conceptual network causes loss of entire concepts, whereas a distributed network loses accuracy, as people do, not entire concepts (Churchland and Sejnowski 1992). Third, symbolic representations are not, in themselves, statistically sensitive. In other words, they are not constructed based on statistical regularities found in the environment. Therefore, symbolic representations are not amenable to modeling low-level perception. This sort of low-level perceptual learning seems to be common to all animals, and is an important part of human development. It is the case, however, that though symbolic representations are not statistically sensitive, they are superbly structurally sensitive. To many, this had seemed a reasonable trade-off in light of structurally insensitive alternatives. However, with the more recent development of distributed representations that exhibit both structural and statistical sensitivity, this trade-off is no longer as justifiable.
 
With these limitations in mind, many cognitive scientists have found distributed representations very appealing. Though distributed representations are a far less intuitive form of representation, the advantages they provide easily outweigh the difficulty in initially understanding them. In order to minimize a reliance on a technical description of distributed representations, I have constructed an analogy which will aid in explaining distributed representations in general. The analogy is this:
 
       Imagine all of your concepts are mapped to the surface of a sphere - a conceptual golf ball, if
       you will. Each concept is nestled in a dimple on surface of the golf ball. The more similar two
       concepts are, the closer together they will be on the surface of the golf ball. Also, the
       distance from the center of the golf ball to any particular dimple (i.e. concept) is
       approximately the same.
 
The easiest way to identify the position of a concept on the surface of the ball is to provide its coordinates. These coordinates are in the units of "golf ball radii" and are continuous real values. Such a set of coordinates is commonly referred to as a vector. In order for us, as symbol users, to distinguish this vector from others, we assign it a tag e.g. ‘cherub’. Thus, the vector (0.5, 0.5, 0.707) may be used to represent the position of the 'cherub' dimple on the surface of the conceptual golf ball. Such a representation of 'cherub' is a distributed representation, because the 'cherub' dimple can only be located through knowing all three values. Thus, the concept of 'cherub' is shared, or distributed, across the three dimensions of the golf ball's coordinate system.
 
Notably, a dimple on a golf ball is not a single point, rather it is a collection of points which are in proximity on the surface of the sphere. This makes the dimple particularly appealing as an analog for a concept for the following reasons:
 
 
       1.A prototype for a concept could be considered to be the center of the dimple. 
 
       2.Multiple vectors define the dimple/concept's boundaries; thus it is a collection of examples.
          Typically, these ‘boundary’ concepts would be specific examples which the cognizer had
          observed from its environment.
 
       3.Because of 2, the prototype for a concept would not be defined by one particular example,
          but rather a superposition of all available examples of the concept.
 
However, as with any analogy, there are a number of issues which may be obscured or inadequately addressed by the conceptual golf ball analogy. For instance, when equating concepts to dimples, we must realize that it is misleading to think of concepts as being defined by a static circular boundary as a dimple is. Rather, conceptual boundaries may be dynamic, oddly shaped, "fuzzy", and possibly not even contiguous. Furthermore, such boundaries do not delineate two or three dimensional figures but rather some n-dimensional figures where n is possibly in the thousands or even millions. As we are equating a three dimensional golf ball to an n-dimensional hypersphere, it is important to bear in mind the limitations of such a simplification. In the introduction to this discussion, I noted that much of the reason researchers were motivated to look for alternate forms of representation was due to the failings of symbolic representations. In particular we discussed the propositional, brittle, and statistical insensitivity nature of symbolic representations. Distributed representations do not fall prey to these short-comings. Rather, distributed representations:
 
       1.Are the natural result of organization of statistical input and thus provide a natural means
          to capturing semantic information (Smolensky 1995); 
 
       2.Have been successfully applied to visual, olfactory, auditory and tactile problems; 
 
       3.Have been proved to degrade gracefully with noise and are commonly tested with
          simulated lesions (i.e. a removal of part of the representation).
 
Furthermore, distributed representations have a number of properties which lend significant psychological and neurological plausibility to models employing them. In many instances distributed representations:
 
       1.Represent concepts continuously; 
 
       2.Can be processed in parallel; 
 
       3.Can be learned using proven methods (Hinton 1986).
 
These powerful properties, coupled with the ability of distributed representations to overcome important short-comings of symbolic representation provide a solid foundation for realistic computational models of human cognition.
 
 
Chris Eliasmith