GPU Programming Project 1: GLSL Webcam shading

Downloading and running program
To download the src and executable for Mac go here.  

To compile from source, you must first install OpenCV and glew.  (I recommend using MacPorts for this)

YouTube Video

My GLSL webcam shading program uses OpenCV to load a video feed from a webcam to an OpenGL program and then modifies the base image using shaders written in GLSL.  I have three pages, each of which show the normal video feed and three custom shaders.  To view one shader at a time, type ''q' to view the upper left image, 'w' to view the upper left image, 'a' to view the lower left image, or 's' to view the lower right image.   To move between pages, type '1', '2' or '3'.  

Page 1, Shader 1:  Cartoon

This shader adapts the video feed to produce a cartoon or comic book look.  To accomplish this, I used the following methods
        1) Blur using a gaussian-filter convolution kernel 
        { 1/16 , 2/16 , 1/16
         2/16 , 4/16 , 2/16
         1/16 , 2/16 , 1/16 }
        2)  Emboss image, multiple directions
               { 2 , 0, 0
         0 , -1, 0
         0 , 0, -1 }
       The resulting edges are colored black , to give an outline look.     
    3)  Posterize by converting the continuous range of colors into discrete groups
        4)  Brighten, with differential degrees of brightening for red, green and blue

Challenges in developing this shader:  
It is difficult to control edge drawing.  Variations in luminance enhance this problem, resulting at times in undesirable lines.  Similarly, it was difficult to generate a color scheme that looked like a cartoon scene for different lighting conditions.

Overall, however, the shader effectively makes the images look as if they were drawn or sketched by an artist. 


Page 1, Shader 2: Lite Brite

This shader attempts to transform the scene into a 'lite-brite' creation-  discretizing the video feed and transforms each circular unit in the image into a small colored light bulb.  The shader does this by
        1) Procedural shading to place each pixel in a pin or the border between pins.  Pins in odd rows are offset, as in the Lite Brite toy.    

        2)  Compute the average color within the pin.  
    3)  Assign Lite Brite colors (white, yellow, orange, pink, red, purple, blue and green) to pins according to the pin's average saturation and value.  Pins that had low saturation or high value were assigned to the background, since this better permitted features to be defined.  
        4)  Brighten, with differential degrees of brightening for red, green and blue

Challenges in developing this shader:  
Mapping the continuous range of colors to lite-brite colors was a considerable challenge.  Many of my attempts resulted in either a noisy, ill-defined scene.  For this reason, feature definition tends to come from the 'black'/non-pin space.  This, however, is consistant with many of the real-world lite-brite creations I have seen.  


Page 1, Shader 3: 70's, low budget film

This shader attempts to capture the qualities of a low-budget film from the 70's and 80s- grainy, blurry with jumps in frames and other inconsistencies.  The shader does this by
        1) Gaussian blur, as described above 

        2)  Animated brightness- I passed the time to the shader and then varied the brightness of the scene according to a sinusoidal function.  

        3) Frame jumps- every thirty seconds or so (exact timing is variable) the scene jumps in x or y.  

        4) Film grains- at random points black spots appear and disappear.  Also, horizontal black lines occasionally disrupt the picture.   

Challenges in developing this shader:  
I wanted to blur the picture even more.  To do this, however, I need multi-pass shading, a technique I could not implement in this project. 


Page 2, Shader 1: Image Distortion and Chromatic Abberation 

This shader spatially and chromatically distorts the image, as if seen through a polarizing and curved lens.  To complete this shader, I used the following as references:

        1) Convert the image into space ranging from 0 to 1

        2)  Apply a spatial distortion based on the following function:  

                float k  = -1.0;//-0.15; //distortion coefficient

                float kcube = .5;//.15; //cubic distortion value

                float scale = .9;

                float dispersion = .01;

                float blurAmount = .5;

                float r2 = (x- .5) * (x-.5) + (y - .5) * (y-.5);

                //compute cubic distortion, if necessary

                float mult = k;

                if( kcube != 0.0 )

                    mult = k + kcube * sqrt( r2 ); 

                float f= 1.0 + r2 * mult;  //factor of distortion 

        2)  Apply a chromatic distortion.  For r, g and b a distortion value is computed, and then the r, g or b component of the image at its distorted postion is red.  So, if               
strong red distortion is applied, texture look up for the red value will be shifted to a greater degree than that for a green or a blue value.   


Page 2, Shader 2: Bump Map

A bump map takes an image as input, converts it into a 'normal' map to alter light reflection off the surface.  This should create a textured or 'bumpy' look.  To do this:

1) Vertex shader must pass vertex light and normal positions, which will be linearly interpolated across the fragments

2) Fragment shader reads from input image texture and uses several emboss kernels to determine the light reflection at that point.  I used the embossing result as a normal map, modulating the normal in proportion to the luminance of the pixel after embossing.  

Challenges:  I believe this effect would best be achieved through multi-pass shading, an approach I was unable to implement for this project.  However, it is clear that light reflections are modulated by embossed edges.  However, the surface does not look as 'bumpy' as I would like.  


Page 2, Shader 3: Fun Mirror

For this shader, I made a dynamic 'fun house mirror', whereby the image distortion varies over time according to a sinusoidal function.  In the shader, I did the following:

1) Converted to texture space to coordinates -1 to 1, then converted to polar coordinates, in terms of r phi

2) Modified the image radius by a function in terms of the time since program start

    float t = mod( float( timer ), 30000.0 ) / 1000.0; //0 - 30 seconds 

    float fac = abs( sin( t ) )*2.0 + .125;

    r = pow( r, fac );//fac );

 To complete this shader, I used the following as references:  

For this shader, I decomposed the image into disjoint rectangular regions, and then rearrange them in a final, multi-color collage.  The primary work was performed in the fragment shader.  Each frame pulls from a different portion of the texture.  Scaling varies between frames as does color amplification varies between frames.  Placement of frames, along with scaling and color mapping was not automatically computed.  I wanted to create a specific effect, that is of pulling from the central portion of the image and reconstituting with slight variations in each frame.  This took a bit of extra work, but I am pleased with the result.  


Page 3, Shader 2: Dynamic Crystal

For this shader, I used a geometry shader to create a dynamic, multi-colored crystal effect.  This shader was completed by doing the following:

1) Breaking the original image into 3 triangles. 

2) Each vertex on the rectangle was given a unique color- yellow, blue, orange and pink.   

3) Passing the triangles and colors through the vertex shader to the geometry shader.

4) For each triangle, create 9 additional triangles going from the midpoints of each line segment to the center of the triangle to each of the vertices.  
5) For each new vertex, warp the texture coordinates and color mixing according to a set of sinusoidal functions in terms of time.  This creates the time varying distortion and color shifting. 

The result is what I would imagine you might see if viewing the image through a dynamically changing crystal.  


Page 3, Shader 3: Wave

For this shader, I used a geometry shader to create a wave effect.  To create this shader, I used the following reference:

This shader works by 
1) Shifting the coordinates so that the texture look-up goes from -1 to 1

2) Compute the distance from the center of the wave, and use that to determine where this fragment is within the wave

I had to tweak the wave function from the website rather substantially to get wave amplitude and frequency to an ideal point.  In addition, the center of the wave is modulated by movement of the mouse.