Embedded Files
Fig. 4.1.1
Color analysis is important for many image processing initiatives. To gain familiarity with color representation in the digital realm, we worked through the exercises detailed below. Our color detection algorithm can be used to isolate objects of a particular color, determine whether an image has a significant presence of a certain color, and quantify the "distance" between two colors, as we demonstrate below. Such algorithms have numerous uses in applications related to image segmentation and object identification.
Our objective in this task is to develop an algorithm that reads an image as input and outputs
a logical value (0 or 1) indicating whether there is a significant presence of the red color in the image and
a processed version of the input image showing only the identified red pixels in the input image.
Theoretically, the methodology in our algorithm (the "distance comparison" algorithm, as discussed below) is general enough to handle any color of our choice, but for the sake of concrete illustration, we will demonstrate our journey with the development of this algorithm with the assumption that we are interested in the red color of this cloth on the right.
Fig. 4.1.2
After preprocessing the data, the development journey of our algorithm can be roughly divided into four stages:
1) come up with a way to determine whether a pixel is considered red or not,
2) count the number of red pixels in the image and compare it against an experimentally determined threshold to classify whether the image contains a red object or not,
3) window the input image to observe which pixels in the image are classified as red, and
4) repeat the previous three steps and test the algorithm on different data until a satisfactory performance is obtained.
As a whole, we developed two algorithms for this color detection task, and they primarily differ in stage 1. For ease of reference, we will call the first algorithm we developed "contrast adjustment and intensity comparison" algorithm and the second one "distance comparison" algorithm. The demo of our "distance comparison" algorithm can be found in this GitHub repository.
"Contrast adjustment and intensity comparison" algorithm
The “contrast adjustment and intensity comparison” algorithm is inspired by the example in the MATLAB tutorial (Getting Started with Image Processing Algorithms for Parrot Minidrones). The basic idea is that we compare the intensity value of the red channel with the values of the green and blue channels.
Let RGB be an image. As discussed in the preprocessing step, this variable has unsigned-integer-8 type and size 120 x 160 x 3. For each of the 120 x 160 = 19200 pixels, our algorithm for determining whether to call it a “red pixel” is to compare the intensity value of the red channel the sum of the halves of the intensity values of the green and blue channels. More precisely, for all i from 1 to 120, for all j from 1 to 160, the (i, j) pixel is considered a red pixel if RGB(i, j, 1) is strictly larger than RGB(i, j, 2) / 2 + RGB(i, j, 3) / 2 + 100, where 100 is an arbitrarily determined threshold value. We can then sum up all the logical values to count the total number of red pixels. If the total number is strictly larger than 2000 (which is another arbitrarily determined threshold value based on trial and error), we would call the entire image represented by RGB a “red image.”
The problem with the approach above is that a lot of the pixels that should be considered red visually aren’t really determined to be red by the algorithm. To alleviate the problem, we utilize the imadjust() function from MATLAB’s Image Processing Toolbox. This function first “saturates” the RGB intensity values of the input image (for each of the color channels, all pixel values that are below the lower threshold are set equal to the lower threshold, and all pixel values that are above the upper threshold are set equal to the upper threshold) and then “stretches” them to cover the entire range of the unsigned-integer-8 data type (from 0 to 255). This essentially adjusts the color contrast in the image, and we can take advantage of this to essentially make the algorithm more “sensitive” to red pixels.
To summarize, the first step of the overall algorithm is to adjust the contrast of the image. As shown below, the image after contrast adjustment (shown on the right) is significantly “redder” than the original image.
Fig. 4.1.3
The next (and final) step is to classify each pixel as red or nonred according to the intensity values of the three color channels of the adjusted image. If there are a total of more than 2000 pixels (this threshold value corresponds to about 10% of the total image area), then the whole image is determined to contain a red object. We note that this is our initial development of the red color detection algorithm, so the threshold value of 2000 is somewhat arbitrary; in the "distance comparison" algorithm below, we use 5% of the total image area as our threshold value instead.
Fig. 4.1.4
Fig. 4.1.5
The test result of this algorithm using the data of red images is shown above. It appears that the algorithm successfully classifies all the red images as red images and all the nonred images as nonred images. However, when we test it on more diverse image data, we found that the algorithm is overly sensitive to red colors (nonred images are classified as red images). We tried solving the issue by tuning the "contrast adjustment" parameters and the "intensity comparison" parameters, but it never led to consistently satisfactory results. This motivates the development of another "red-pixel detection" algorithm, namely the "distance comparison" algorithm.
"Distance comparison" algorithm
It is not hard to intuitively make sense of why the “contrast adjustment and intensity comparison” algorithm is not perfect. During the “contrast adjustment” step, we deliberately “distort” our image data in some sense just to make our overall algorithm more sensitive to red colors, while in the “intensity comparison” step, we make a somewhat baseless assumption that the intensity value of the red channel of a red pixel is higher than the half of the sum of the intensity values of the other two channels by at least a certain amount. Thus, we seek to develop another algorithm that not only preserves the integrity of the input data but also is more theoretically sound.
Inspired by the concepts of Hilbert spaces and Euclidean distances discussed in lecture, we researched various distance metrics for comparing two different colors. Ultimately, we settled on the “redmean” approximation (Color difference - Wikipedia).
Let RGB be an image. Let R_2, G_2, and B_2 be the reference red, green, and blue intensity values. For all i from 1 to 120, for all j from 1 to 160, if we let R_1 (RGB(i, j, 1)), G_1 (RGB(i, j, 2)), and B_1 (RGB(i, j, 3)) be the red, green, and blue intensity values of the (i, j) pixel, then the distance (denoted by delta_C) between the color of the (i, j) pixel and that represented by R_2, G_2, and B_2 is calculated as follows:
Fig. 4.1.6
Fig. 4.1.7
With the distance metric established, the next question left is what the reference values should be. Fortunately, this question is intuitive to answer. Since we can visually identify the red pixels of our red image data, the reference values can be determined as follows. First, we use MATLAB's Image Segmenter App to segment out the red portion of the image by hand (i.e., visually). We then take the average values of the intensity values of the three color channels to be the reference red, green, and blue values of the red portion of the image. Distances are then measured from these values using the redmean approximation.
Fig 4.1.8
Fig 4.1.9
Now, in order to classify an arbitrary pixel as red or nonred, there is one more element that we need: the distance threshold. We need to compare the calculated distance of the pixel to an established threshold in order to have the algorithm determine whether the pixel should be considered red or not. We determine this threshold using three forms of evidence: histograms of distances of pixels for a typical red image and for a typical nonred image, visualization of the red portion of the distance-based segmented typical red image, and visualization of all the colors that are classified as red.
Fig 4.1.10
Fig 4.1.11
For a typical red image, for the distance range of 0 to 200, the approximate trend is as the distance increases, the number of pixels appears to first increase, then decrease, and then increase again. If we think of the number of pixels as a function of the distance, then the function reaches a "local minimum" at around a distance of 100.
For a typical nonred image, on the other hand, for the distance range of 0 to 200, the approximate trend is the number of pixels remains zero, then increases starting at a distance of about 80, and then decreases. If we think of the number of pixels as a function of the distance, then the function reaches a "local maximum" at around a distance of 140.
Based on these two observations, we might hypothesize that a distance threshold of between 80 and 100 would be a reasonable one for classifying a pixel as red or nonred. Indeed, with a distance threshold of 80, the number of red pixels in the distance-based segmented image is roughly the same as that in the visually segmented image (with the Image Segmenter App).
The choice of our distance threshold (being 80) can also be evaluated by visualizing the pixels in the sample red image that are classified as red:
Fig 4.1.12
Since it is likely for a pixel next to a red pixel to also be a red pixel, it is reasonable to tell the algorithm that the nonred pixels that are "surrounded" by red pixels are misclassified and that they should also be red. Likewise, when a red pixel is "surrounded" by nonred pixels, it is likely that that pixel is misclassified and that it should be nonred. Therefore, to accomplish both of these adjustments, we use morphological filters to determine the "adjusted red pixels:"
Fig 4.1.13
This visual comparison shows that the distance threshold of our choice (80) would yield an algorithm that can identify the vast majority of the red portion of an input red image.
Finally, we visualize all the colors (out of 256 ^ 3 = 16777216 colors) that are classified as red by our algorithm:
Fig 4.1.14
Even though there are more than 600 * 600 = 360000 colors that would be classified as red by our algorithm, they all look roughly the same to a human eye. However, upon closer examination, we see that as we move from left to right in the figure above, the color looks "less and less red" (and maybe "more and more noisy"). Indeed, in creating this figure, we sorted the array in ascending order of distances, resulting in pixels on the left generally having smaller distances then those on the right.
In addition, one could see that even though we call our task "red color detection," the actual color we are detecting looks more like pink. Since our drone's camera is not perfect, the actual red color of our object unfortunately looks like pink in the picture captured by the drone. However, we can alleviate this issue by taking more sample data of red images and add "alternative" red, green, and blue reference values into our algorithm.
Fig 4.1.15
Fig 4.1.16
Fig 4.1.17
To summarize, our "distance comparison" algorithm works as follows. First, for every pixel in the image, we calculate the distance of its color from the typical red color as well as the distances of its color from the alternative reference colors. Then, if at least one of these distances is below our distance threshold (80), then the pixel is classified as red. After performing the classification of all the pixels, we count how many red pixels there are in the image. If this number turns out to be at least 5% of the entire image area (0.05 * 120 * 160 = 960), then we declare that there is a significant presence of a red object in the image and we classify the image as red, otherwise we classify the image as nonred. We show the demonstration of the performance of this algorithm below.
Performance demonstration
Below, we demonstrate the performance of our "distance comparison" algorithm on various image data. We note that we have tested our algorithm on hundreds of images, but for the purpose of illustration, we will only show a select few that cover a broad range of cases.
Fig 4.1.18
Sample image 1 contains a red cloth and is the example we used above. Note that the processed sample image contains slightly more red pixels than the "sample image with identified and adjusted red pixels" in the discussion above does. This is because in producing the plot in the discussion above, there is only one reference color, whereas in our final color detection algorithm, there are three.
Fig 4.1.19
Sample image 2 is a slightly modified version of sample image 1, where there is a hand in the image. The processed sample image shows that the pixels of the "hand portion" of the image are correctly classified as nonred for the most part.
Fig 4.1.20
Sample image 3 contains a red plate and is used to produce an "alternative red color" in the discussion above. The processed image shows that the vast majority of the plate can be correctly identified by our algorithm. The only reason for the missing portions on the lower right is the presence of glare.
Fig 4.1.21
Sample image 4 contains the same object that sample image 3 does, but due to the different lighting condition, the red color here looks slightly brighter than that in sample image 3. However, the processed sample image shows that our "distance comparison" algorithm is still able to successfully detect the red pixels of this image.
Fig 4.1.22
Sample image 5 contains some red color on a touchscreen and is used to obtain the second "alternative red color" in the discussion above. The processed image shows that while the red portion of the image is nicely segmented from the input image, our algorithm is also classifying some yellowish color from the glare on the top left as red.
Fig 4.1.23
Sample image 6 is a different version of sample image 5. The red color in this image looks slightly lighter than the one in sample image 5, but, as shown in the processed image, our algorithm is still able to identify those red pixels. However, we, again, see the effect of glare on the performance of our algorithm.
Fig 4.1.24
Sample image 7 does not contain any red object. We include this here to show that for an image that is objectively nonred, our algorithm can generally successfully classify the image as nonred. Since no red pixel is identified in the input image, the processed image is all-black. We note that our previous algorithm ("contrast adjustment and intensity comparison" algorithm) would falsely classify this image as red, which shows the superiority of the "distance comparison" algorithm.
Fig 4.1.25
Sample image 8 is also a nonred input image. However, as in sample images 5 and 6, with our settings of the reference red colors and the distance threshold, our algorithm would also classify some yellowish color as red. This is evident in the processed image here. Nevertheless, this issue is not too problematic, because the number of (mis)identified red pixels is far less than our area threshold (5% of the total image size), resulting in the correct determination that there is no significant presence of a red object in the input sample image (as indicated by the containsRed output variable). We note that our previous algorithm ("contrast adjustment and intensity comparison" algorithm) would falsely classify this image as red, which shows the superiority of the "distance comparison" algorithm.
In summary, our color detection algorithm (the "distance comparison" algorithm) is largely successful in the sense that it can generally successfully detect all the red pixels that we would expect it to detect (as in sample images 1 through 6). Moreover, the design of our algorithm is such that we can generalize it to detect any reference color of interest.
The top causes of a failure to detect a red pixel is the presence of shade (as in sample images 1, 2, and 4) or the presence of glare (as in sample images 3, 4, and 6). The presence of shade makes the pixel appear darker and look more like black, whereas the presence of glare makes the pixel appear similar to white. The top cause of falsely classifying a pixel as red is the presence of a yellowish color. This color can come from the glare effect (as in sample image 5), or it can come from the object itself (as in sample image 8). Lowering the distance threshold in our algorithm would alleviate this issue.
Page updated
Report abuse