Colors in the Brain project

What happens in our brain when we see colors?

We perceive colors naturally when we open our eyes. It is a sensation that happens mainly in our brain, rather than in eyes. However, we know little about how the signal of colors are processed in our brain. What kind of cells are working in which way, and what kind of neuron's signal enables our color perception? That is the aim that we want to clarify through this project.

Apology on low quality English writing by IK.

This HP is to introduce the issues that we want to clarify and approaches to clarify them in our project supported by category (A) JSPS KAKENHI (Grant-in-aid for Scientific Research No. JP20H00597, from Apr.2020 to Mar. 2024).

Table of contents (you can skip collapsible "boxes").

The very first stage (cone photoreceptors)
The first calculations to see colors (cone-opponent mechanism)
Color and words (color categories)
Subtle difference in shades of colors ("color appearance") and brain
Target of our project

1. The very first stage (cone photoreceptors)

Human eyes have three types of optical sensors, called "cone photoreceptors", in the retina.

They differ in its sensitivity to the wavelength of light and are called short-, medium-, and long-wavelength sensitive cones; in short S-, M-, and L-cones.

They are differently activated by receiving light (photons) of different wavelength. But the cone themselves do not code color appearance.

Four panels of images on the left demonstrate what I mean that cones do not code color appearance by itself. Three black-and-white images are simulating the intensity of responses of each cone types (L-, M-, and S-cones) to the full-color image on the top left.

Since cones simply response to the intensity of light they can receive, depending on the sensitivity to each wavelength (see graph in the image of previous paragraph), they simply respond in the amount of energy they receive from light.

Therefore, the blue sky appears brightest in S-cone response image, because it responds most to short wavelength (bluish) lights, and the orange hut on the green slope appears slightly brighter in L-cone image, because it responds slightly better to longer (reddish) light than other cone types.

Such subtle differences in cone responses code colors.

Box 1. What are colors, in the first place?

This collapsible box introduce some side stories, that are not necessary to follow the main story. Please take a look, if you like.

What are colors, in the first place? Have you ever thought about it?

In daily basis, it is not always vital to see colors to recognize objects around us. For example, we can roughly recognize objects in black-and-white pictures / films. We can tell objects' material, such as glossiness, transparency, or whatever, without colors. We can sway to avoid collision of objects that suddenly appears in the corner of the eye and approaching rapidly toward you, without perceiving colors. Indeed, human visual system to detect object shape, motion, stereopsis, ... are independent from color-processing neurons. They are not sensitive to colors.

On the other hand, in order to recognize the status of objects, we often need colors. For example, the ripeness of fruits or vegetable can be estimated by its color. If we fail to recognize it, we may become sick and may not achieve energy to live. Or, as we human is a social animal, we need to detect the emotional state of opponents by their facial color expressions. We definitely need colors in those scenes. In addition, we feel better/happy when we change the color your room walls. Sometimes, we feel happy to see scenic place with beautiful color contrast, like autumn foliage of mountains. Recent AI technology makes it possible to colorize old films/photographs, and you may feel that colorized scenes are very live and recent than actual. So, colors seem to affect our sensation of reality in some way.

Therefore, the colors are important to our survival and sometimes strongly effective to our emotional states. It is a very interesting and miraculous sensation.

Now, let's take a look at some physical aspect of colors. When we see different wavelength of light, we perceive different colors. Colors in rainbow is a result of refraction of sunlight when split by a prism-like effect, which occurred from the presence of water particles in atmosphere by rain or fog. Because the direction after refraction differs by wavelength of light, the sun light splits and ordered in its wavelength from blue to red in the spectrum.

Then, do we see different colors whenever the wavelength is different? It is not always true. When we bring a piece of white paper under the sunlight and under incandescent lamps or candle light, we perceive same "white" from that paper. Some people may find that the shade of whiteness are slightly different; the color under candle light may appear to be tinted slightly in amber. However, such difference in appearance is much smaller than actual difference in the wavelength of light reflected on the white paper. This phenomenon is called color constancy. You can also find optical illusions in which different RGB colors yield same color, or vice versa.

These phenomena implies that the wavelength of light and color appearance are not corresponding in 1-to-1 manner. Some readers may be confused by this fact, but the reason why we see various colors in rainbow is because we perceive them at the same time; I mean, the visual environment is the same to find blue and yellow bands in the rainbow. The "white" paper example occurs when the lights surrounding the observer are different; one under sunlight and the other under candle light and the states of adaptation of our vision are different (and some other factors are working, but I skip them for simplicity).

It means, colors (particularly its appearance) have no physical measures or rulers, different from length or weights. But we have very confident sensation of color in ourselves. This is one of the uniqueness of color perception and attracted famous physics "giants" like Eissac Newton, Herman von Helmholtz, Erwin Schrödinger, and etc.

(end of Box 1)

2. The first calculations to see colors (cone-opponent mechanism)

As the sensitivity curves for two photoreceptors (L- and M-cones) are very close (red and green lines), responses of these cone types are usually very similar. The black-and-white images in the second panel clearly demonstrates it. Therefore, such small differences could be "messed up" by neural noises that will come in on the long way from retina to brain, if you try to bring each cone response by an independent line (neurons) in its crude form.

So it is necessary to take the difference of these photoreceptors and enhance, before sending them to brain. The cone-opponent cells (or simply "opponent cells") plays a significant role in taking differences and enhancement before sending the color signal to brain. The opponent cells were found in 1950s and the differences of L–M and S–(L+M) seem to present in human retina. L–M difference roughly carries red-green color differences and S–(L+M) carries blue-yellow differences (also, see diagram with triangle and circles connected with lines, in the top panel).

If you take these two opponent-cell outputs as axes of 2D cartesian coordinates, you may find that a point (coordinates) in this 2D plane represents a color. This plane is called opponent-color space. Differences in the kind of colors (such as red, pink, purple, and etc.) are called hues and the distance of the point from origin represents vividness or color contrast.

Box 2: the story of trichromacy and opponent-color theory

At the end of 19C, two physics scientists Thomas Young and Herman von Helmholz found that the combination of three primary colors (e.g., red, green, and blue) can render any colors that human can see and they proposed a theory of trichromacy, in which human vision has three primary mechanisms for color sensation. This principle of color mixture is widely used in color displays nowadays in computers, smartphones, and etc.

Same time, a psychologist and physiologist Ewald Hering had found that human rarely calls red and green sensations and blue and yellow sensations at the same time, and the long-time observation of red (blue) yields green (yellow) afterimage. He considered that these two pairs of colors are antagonistic and constitutes fundamentals of color sensation, and proposed opponent color theory. It is also proposed that any color can be represented as the mixture of two of the four landmark colors: red/green and yellow/blue. For example, orange can be represented as the mixture of 50% red and 50% yellow; this method is called elementary color naming and is still used as a method to study color appearances in basic research.

The debate between these two theories seem to reach to a happy end by the finding of three cone types to support trichromacy and the finding of opponent cells supports opponent-color theory. However, ...

Please follow the story after this after this box. :)

(end of Box 2)

Then, is color appearance decided in retina? It is not true.
We find colors that appears pure red, blue, green, or yellow: they are called unique hues. For example, we can determine a unique yellow by finding a color that appears neither reddish nor greenish.

The diagram by Webster et al. (2005) found that the distributions fo unique hues in the opponent-color space do not coincide with the axes that corresponds to the pure outputs of opponent cells. Each dot shows individual unique hues of 50 participants, but most dots (except unique reds) are not aligned with the axes. Such individual differences can not be explained by the variability of optical system or eyes or ratio of L-, M- cone population in the retina.

It means, color appearance is not defined in retina.

To summarize the story, so far, information of colors are captured by the three cone types and extracted by opponent cells to emphasize the subtle differences of cone responses before sending to brain, that are not linked to color appearance. These are hard-wired and automatic processes, and the signal after opponent cells are submitted to brain.

Then, where in the brain is responsible for color appearance, given that opponent cells do not define color appearance? If the signal of neurons that represent color appearance were different from opponent-cell outputs, what kind of neurons represent our color appearance, and in what kind of neural signal.
The clarification of these questions are the ultimate target of our project.

3. Color and words (color categories)

Let's take a look at this problem from a different view point.

We describe colors by using color terms. We can refer to colors of slightly different shades by a color term. Such groups of colors are called color categories.

This diagram is a color chart in studies of color terms. The tile of small color chips of different hues and lightness are bordered by lines for 11 basic color terms: red, green, yellow, blue, purple, orange, pink, brown, white, gray, and black.

We use such color categories quite frequently in daily life. The color categories are known to code colors that are processed in brain areas close to the process of object recognition and memories.

4. Subtle difference in shades of colors ("color appearance") and brain

When we perceive colors, we find continuous, gradual changes in color that can be categorized in the same group or can be represented as a color term. On the contrary, such color sensation is not represented by categorical color perception.
When we notice, for example, a reddish brush of face in cheek, we do not perceive red as a color category that includes red for traffic signal; we rather perceive a slight shade of red in face. Such color perception is not based on the opponent-cell responses in the retina.

Then, in which form such different shades of colors, called color appearance, are represented in human brains?

One of the candidate is neurons that responds to particular shades of colors, found in macaque brain in electrophysiological studies. We consider that these color sensitive neurons may take a part in the representation of color appearance in the human brains.

In one of our previous studies, we found that neurons that are responding more to the intermediate directions of opponent-color space (e.g, purple, cyan, or orange) in functional MRI (fMRI) studies in collaboration with a team of researchers in RIKEN (Kuriki et al., 2011; Kuriki et al., 2015)．

This diagram shows the histogram of responses to each direction of colors (shown schematic colors in a circular band) in human primary visual cortex (the primary locus where signal from eye reaches to brain), by the distance from the center (Kuriki et al., 2015). This result shows that population of fMRI responses distribute not only in the direction along horizontal and vertical axes, but are found in intermediate directions.

We consider neurons that evoked such fMRI signal may be relevant to the neurons found macaque brains, and are playing a part of the roles in the representation of color appearance.

Box 3: precise interpretation of our fMRI results

The diagram represents the number of voxels (i.e., volume element of images of fMRI signal) in a radial histogram. Each voxel contains a large number of neurons (approx. tens to hundreds of thousands). If neurons sensitive to one hue and neurons sensitive to its opposite hue are contained in a voxel, the fMRI signal from the voxel becomes null.

For example, population along horizontal axis which corresponds to L–M output of retinal opponent cells, are almost none in the diagram. However, various electrophysiological studies in macaque visual cortex exhibit a dominant responses of L–M opponent responses. The presence of roughly same number of neurons sensitive to positive (reddish) and negative (greenish) hues of L–M responses could have caused such a null population in our result.

It means, the result of our fMRI study (Kuriki et al., 2015) may be a tip of iceberg. To get rid of this problem, we started another study using a kind of EEG (electro-encephalograpy; "brain waves") responses called SSVEP.

( end of Box 3)

5. The direction of our project

In our KAKENHI project, we will record cortical responses by fMRI and EEG to compare them with the perception of color appearance, measured by psychophysical methods, in an attempt to elucidate how color appearance is represented as neural signal in human visual cortex.

Such color sensation to different shades of colors could be an innate one or could develop through the aging of children. To clarify this point, some member of our group (led by Prof. Yamaguchi at Chuo Univ.) will investigate cortical responses from infants.

The locus of brain activity can be better specified by fMRI, and fMRI studies will be carried in collaboration with fMRI support unit of Center of Brain Science, RIKEN, led by Dr. Ueno.

As overseas collaboration, we will also work with Professors Søren Andersen (University of Aberdeen, U.K.) and David H. Peterzell (Fielding Graduate University, U.S.A.) on analyses of individual differences in SSVEP (see Box 4) amplitudes for various hue.

Our ultimate goal is to clarify signal transformation from opponent-cell outputs to color categories through color appearance representations.

Box 4: A study using SSVEP

When we present a flickering light in the observer's visual field, it evokes oscillatory brain wave at the same temporal frequency as the light flicker. This component of brain response in EEG is called SSVEP (steady state visual evoked potential). The SSVEP reflects the activity of neural mechanism that is related to the processing of flickering light.
So, when we vary the color of this flickering light, the SSVEP amplitude reflects the strength of neural process in the participants' brain. It may reflect the difference in the sensitivity or population of neurons sensitive to each hue.

Our first paper on this study has been published recently.
Kaneko S, Kuriki I, & Andersen SK. (2020) Steady-state visual evoked potentials elicited from early visual cortex reflect both perceptual color space and cone-opponent mechanisms. Cerebral Cortex Communications.

We found remarkable individual differences in the pattern of SSVEP amplitudes to hue differences, and we are now investigating the mechanisms that is responsible for the SSVEP responses utilizing this individual differences.

(end of Box 4)

Box 5: A story about the study of color and its brain function

Color is a very different sensation from shape and size of objects; we can not confirm the sensation by touching objects. So, it is also difficult to share the sensation with a different person. Then, do people have same sense of color to the same object? This is a very difficult philosophical issue.
Since color appearance is a subjective sensation and difficult to express everything about color appearance, even by using hundreds of words.

It is possible to share the sense of color identity. You can agree to other person about the similarity/difference of colors. It is just sharing a part of the color appearance. Therefore, color appearance is a unique sensation whose process is entirely closed within a person's brain after it enters from eye.

Therefore, the clarification of color information processing is one of the essential issue in the understanding of visual information processing, as a while. We expect that the clarification of color information processing may also be a help of understanding other information processing mechanisms in human brain.

(end of Box 5)

Box 6: Another issue on individual differences in color sensation

The presence of individual difference in color sensation is very obvious. We find different kind of color more suitable than other, but other person find different color as a better one. We buy cloths or cars in different color. On the other hand you may have an experience of finding a friend who has similar preference/taste of colors. It implies the processing of color information is common in some part and different in other part.

We found a lot of individual differences in the response of visual cortex to hues (Kuriki et al., 2015; Kuriki et al., ICVS, 2019; Kaneko et al., 2020). Color appearance measurements also show remarkable individual differences as shown in a map of unique hues (Webster et al, 2005) in a panel above. It is also known that other person's unique hues can't be a unique hue of yours.

However, we have not found how these differences in brain activity are related to individual color sensation. If we could find their correspondence, we might be able to quantify the preference to color by the measurement of fMRI or EEG responses by non-subjective measure.

In the present KAKEN project, our main target is the common mechanisms that can be the neural basis of individual differences in color appearance. We consider that this project could be further connect to the understandings of individual difference in the sensation of colors, the prototypes of color sensations, cultural or environmental effect of color sensations, origin of such sensations, etc. Therefore, we believe that this project is a very important issue in various aspect of related fields of study.

(end of Box 6)