Embedded Files
Post date: Feb 2, 2012 7:12:59 AM
1. What is this
This project is about to write different shaders using GLSL to make different effects to a live video captured from a webcam with OpenCV.
I finished two pages of shaders. Other effects I acquired during this project are all in the same difficulty level, so I decided not to put them into the third page.
The video I created as a demonstration is here: http://www.youtube.com/watch?v=pG3Vt1MvlwQ
Click here to download the source code. Or you can scroll down to the bottom of this page, you can find source code in attachment.
2. How to Start
You should first install OpenCV, GLEW and GLUT on your computer.
Of course you also need a webcam.
If you are using Windows, just create a new command line project in Visual Studio, add all files into it, compile and run.
If you are using Mac OS X, use makefile to build and run the application from terminal.
3. How to Use
The shaders are separated into three pages (in my case two pages).
- Use keyboard "1" "2" and "3" to jump to certain page you want to check.
- You can recognize which page are you currently in by checking the title of the application window.
In each page, there are 3 different shaded live videos and one untouched video that in the upper left grid of the window.
- Use keyboard "a" "q" "s" and "w" to toggle one of the 4 videos to fullscreen mode. You can also use mouse click to choose videos.
- The name of the effect you are watching is in the title of the application window.
- Use keyboard "z" to go back to the overview mode. You can also just click the mouse to go back.
4. What I did
4.0 Introduction
- Overview:
Control mechanism of this application is omitted here for clarity. If interested, you can refer to the source code.
- Code:
Vertex shader:
varying vec2 TexCoord;
uniform int whichEffect;
void main (void)
{
...
TexCoord = gl_MultiTexCoord0.st;
gl_Position = ftransform();
}
Fragment shader:
uniform int whichEffect;
varying vec2 TexCoord;
uniform sampler2DRect whichTexture;
void main(void)
{
vec4 sum = vec4(0.0);
vec4 part = vec4(0.0);
...
gl_FragColor = sum;
}
4.1 Two Face
- Overview:
First page, lower left.
It's just a warm up color modification effect.
- Principle:
Simple, just use inverse color in left half of the video area.
- Code:
Fragment shader:
if (whichEffect == 0)
// inverse
{
part = texture2DRect(whichTexture, TexCoord.xy);
if (TexCoord.x < 400.0)
{
part = vec4(1.0, 1.0, 1.0, 1.0) - part;
}
sum = part;
}
4.2 For the Record
- Overview:
Fist page, lower right.
It's a relief sculpture effect.
- Principle:
Compute the difference between the color of current pixel and that of its upper left neighbor.
Convert it to luminance.
Add a gray backgroud color to it.
- Code:
Fragment shader:
if (whichEffect == 2)
// sculpture
{
part = texture2DRect(whichTexture, TexCoord.xy);
vec2 ulCoord = vec2(max(TexCoord.x - 1.0, 0.0), max(TexCoord.y - 1.0, 0.0));
vec4 ulColor = texture2DRect(whichTexture, ulCoord);
vec4 bkColor = vec4(0.5, 0.5, 0.5, 1.0);
vec4 delColor = part - ulColor;
float h = 0.2125 * delColor.r + 0.7154 * delColor.g + 0.0721 * delColor.b;
sum.r = h;
sum.g = h;
sum.b = h;
sum = delColor + bkColor;
}
- Memo:
Reference: http://group.lehu.shu.edu.cn/Article.aspx?aid=147793 (Note: it's Chinese), I changed it from HLSL to GLSL.
4.3 Privacy
- Overview:
First page, upper right.
It's a mosaic effect.
- Principle:
Divide screen to several squares, then use the color of upper left pixel of a square to represent all pixels in this square.
- Code:
Fragment shader:
if (whichEffect == 3)
// mosaic
{
vec2 mosaic = vec2(int(TexCoord.x / 24.0) * 24, int(TexCoord.y / 24.0) * 24);
vec3 color = vec3(texture2DRect(whichTexture, mosaic));
sum=vec4(color, 1.0);
}
- Memo:
Reference: http://group.lehu.shu.edu.cn/Article.aspx?aid=147793 (Note: it's still Chinese), I changed it from HLSL to GLSL.
4.4 Just Like Old Times
- Overview:
Second page, lower left.
It's a (not very realistic) old photo effect.
- Principle:
For each pixel, compute its color by adding its color and its eight near neighbor pixels as well as eight far neighbor pixels, then divide by 17 to get the average color.
Convert it to luminance.
Mix it with sepia color.
Add some 3D noise generated in main application program to it.
Add a frame to it.
- Code:
Vertex shader:
varying vec3 MCposition;
varying float LightIntensity;
varying vec2 NoiseCoord;
vec3 LightPos = vec3(1.0);
if (whichEffect == 4)
// noise
{
vec3 ECposition = vec3(gl_ModelViewMatrix * gl_Vertex);
MCposition = vec3(gl_Vertex);
vec3 tnorm = normalize(vec3(gl_NormalMatrix * gl_Normal));
LightIntensity = dot(normalize(LightPos - ECposition), tnorm);
}
Fragment shader:
varying vec3 MCposition;
varying float LightIntensity;
uniform sampler3D Noise;
if (whichEffect == 4)
// old photo
{
part = texture2DRect(whichTexture, TexCoord.xy);
// original values are 1.3 0.93 4.6 3.25
// I use 2.0 and 20.0 as radius
vec2 l1Coord = vec2(max(TexCoord.x - 2.0, 0.0), TexCoord.y);
vec4 l1 = texture2DRect(whichTexture, l1Coord);
vec2 r1Coord = vec2(min(TexCoord.x + 2.0, 800.0), TexCoord.y);
vec4 r1 = texture2DRect(whichTexture, r1Coord);
vec2 u1Coord = vec2(TexCoord.x, min(TexCoord.y + 2.0, 800.0));
vec4 u1 = texture2DRect(whichTexture, u1Coord);
vec2 b1Coord = vec2(TexCoord.x, max(TexCoord.y - 2.0, 0.0));
vec4 b1 = texture2DRect(whichTexture, b1Coord);
vec2 ul1Coord = vec2(max(TexCoord.x - 1.41, 0.0), min(TexCoord.y + 1.41, 800.0));
vec4 ul1 = texture2DRect(whichTexture, ul1Coord);
vec2 ur1Coord = vec2(min(TexCoord.x + 1.41, 800.0), min(TexCoord.y + 1.41, 800.0));
vec4 ur1 = texture2DRect(whichTexture, ur1Coord);
vec2 bl1Coord = vec2(max(TexCoord.x - 1.41, 0.0), max(TexCoord.y - 1.41, 0.0));
vec4 bl1 = texture2DRect(whichTexture, bl1Coord);
vec2 br1Coord = vec2(min(TexCoord.x + 1.41, 800.0), max(TexCoord.y - 1.41, 0.0));
vec4 br1 = texture2DRect(whichTexture, br1Coord);
vec2 l2Coord = vec2(max(TexCoord.x - 20.0, 0.0), TexCoord.y);
vec4 l2 = texture2DRect(whichTexture, l2Coord);
vec2 r2Coord = vec2(min(TexCoord.x + 20.0, 800.0), TexCoord.y);
vec4 r2 = texture2DRect(whichTexture, r2Coord);
vec2 u2Coord = vec2(TexCoord.x, min(TexCoord.y + 20.0, 800.0));
vec4 u2 = texture2DRect(whichTexture, u2Coord);
vec2 b2Coord = vec2(TexCoord.x, max(TexCoord.y - 20.0, 0.0));
vec4 b2 = texture2DRect(whichTexture, b2Coord);
vec2 ul2Coord = vec2(max(TexCoord.x - 14.0, 0.0), min(TexCoord.y + 14.0, 800.0));
vec4 ul2 = texture2DRect(whichTexture, ul2Coord);
vec2 ur2Coord = vec2(min(TexCoord.x + 14.0, 800.0), min(TexCoord.y + 14.0, 800.0));
vec4 ur2 = texture2DRect(whichTexture, ur2Coord);
vec2 bl2Coord = vec2(max(TexCoord.x - 14.0, 0.0), max(TexCoord.y - 14.0, 0.0));
vec4 bl2 = texture2DRect(whichTexture, bl2Coord);
vec2 br2Coord = vec2(min(TexCoord.x + 14.0, 800.0), max(TexCoord.y - 14.0, 0.0));
vec4 br2 = texture2DRect(whichTexture, br2Coord);
part = part + l1 + r1 + u1 + b1
+ ul1 + ur1 + br1 + bl1
+ (l2 + r2 + u2 + b2 + ur2 + ul2 + bl2 + br2);
part = part / 17.0;
// luminance
vec3 h = mix(vec3(dot(vec3(0.2125, 0.7154, 0.0721), vec3(part))), vec3(part), 0.3);
//float h = 0.3 * part.r + 0.59 * part.g + 0.11 * part.b;
vec3 sepia = vec3(215.0, 172.0, 24.0) / 256.0;
sum = vec4(h * sepia, 1.0);
// resepia
//sum = vec4(mix(vec3(dot(vec3(0.2125, 0.7154, 0.0721), vec3(sum))), vec3(sum), 0.4), 1.0);
//begin to deal with noise
vec4 noiseVec = texture3D(Noise, MCposition);
vec3 sky = vec3(0.36, 0.14, 0.07);
vec3 cloud = vec3(0.5, 0.5, 0.5);
// cloud texture
float intensity = (noiseVec[0] + noiseVec[1] + noiseVec[2] + noiseVec[3] + 0.03125) * 1.5;
// mix sky and cloud
vec3 scColor = mix(sky, cloud, intensity) * LightIntensity;
// mix original and noise
sum = vec4(mix(scColor, vec3(sum), 0.5), 1.0);
//end of noise
if (TexCoord.x > 790.0 || TexCoord.y > 790.0 || TexCoord.x < 10.0 || TexCoord.y < 10.0)
sum = vec4(0.8, 0.8, 0.8, 0);
}
Application code:
void renderScene(void)
{
GLint lightLoc, tangentLoc, noiseLoc;
make3DNoiseTexture();
glGenTextures(1, &noise3DTexName);
glBindTexture(GL_TEXTURE_3D, noise3DTexName);
glTexParameterf(GL_TEXTURE_3D, GL_TEXTURE_WRAP_S, GL_REPEAT);
glTexParameterf(GL_TEXTURE_3D, GL_TEXTURE_WRAP_T, GL_REPEAT);
glTexParameterf(GL_TEXTURE_3D, GL_TEXTURE_WRAP_R, GL_REPEAT);
glTexParameterf(GL_TEXTURE_3D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameterf(GL_TEXTURE_3D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexImage3D(GL_TEXTURE_3D, 0, GL_RGBA,
noise3DTexSize, noise3DTexSize, noise3DTexSize,
0, GL_RGBA, GL_UNSIGNED_BYTE, noise3DTexPtr);
noiseLoc = glGetUniformLocation(p, "Noise");
glUniform1i(noiseLoc, 1);
...
}
int main(int argc, char **argv)
{
...
glGenTextures(1, &noise3DTexName);
glActiveTexture(GL_TEXTURE0 + 1);
}
- Memo:
Reference: https://docs.google.com/View?id=dfmddm8t_291f6sks9cx.
Reference: http://www.ece.uic.edu/~wniedbal/cs525_project1.htm
Reference: http://brianmykietka.net/projects.php?project=glsl
I am not sure about the exact meaning of the description about this effect on the first link, so I contacted Hyejung Hur, the author of this effect, to ask her about the detail of this effect, but I don't think I got the whole of it.
All I can present through this shader is a sepia colored blur video and some stain, couldn't get the realistic 'imprint of age' on it.
As for the frame, I was meant to make it like cinefilm as the picture below, but didn't have time to make it, so I just use a gray rim.
During making this effect, I found an interesting effect, it looks like an old television with two channels conflicting, see the video below:
囧...
At first, I had no idea how I got this effect. After a long time of research and debugging, I found the only thing I did wrong is I commented function make3DNoise() in my code, which leaded to a random texture I think.
(update) Well, no. I comment it again, and now I can't get that effect. What a pity you guys can't see it. I'll try to get it back when free.
4.5 See Me On the Teapot
- Overview:
Second page, lower right.
It's an environment mapping effect.
- Principle:
use the video as an environment map, compute the reflection direction from the teapot to the map to decide which pixel of the environment map will be 'reflected' on that specific spot
- Code:
Vertex shader:
varying vec3 Normal;
varying vec3 EyeDir;
varying float LightIntensity;
if (whichEffect == 6)
// environment mapping
{
Normal = normalize(gl_NormalMatrix * gl_Normal);
vec4 pos = gl_ModelViewMatrix * gl_Vertex;
EyeDir = pos.xyz;
LightIntensity = max(dot(normalize(LightPosition - EyeDir), Normal), 0.0);
}
Fragment shader:
uniform vec3 BaseColor;
uniform float MixRatio;
varying vec3 Normal;
varying vec3 EyeDir;
varying float LightIntensity;
if (whichEffect == 6)
// environment mapping
{
vec3 reflectDir = reflect(EyeDir, Normal);
vec2 index;
const vec3 Xunitvec = vec3(1.0, 0.0, 0.0);
const vec3 Yunitvec = vec3(0.0, 1.0, 0.0);
index.y = dot(normalize(reflectDir), Yunitvec); // altitude
reflectDir.y = 0.0;
index.x = dot(normalize(reflectDir), Xunitvec) * 0.5; // azimuth
// index.x ranges from -0.5 to 0.5
// index.y ranges from -1 to 1
// texture coordinates range from 0 to 1 in both dimensions (aside from wrapping)
// Translate index values into proper range
// t (elevation) value is straight forward, s (azimuth) is a bit more complicated
// if reflectDir.z >= 0.0, s will range from 0.25 to 0.75 (front side)
// if reflectDir.z < 0.0, s will range from 0.75 to 1.25 (-0.25) (back side)
// which is OK since we are wrapping
// t will range from 0 to 1
if (reflectDir.z >= 0.0) // reflecting towards the front
index = (index + 1.0) * 0.5;
else // reflecting towards the back
{
index.s = (-index.s) * 0.5 + 1.0;
index.t = (index.t + 1.0) * 0.5; // same as for index.t above
}
//change index to actual coord
index = index * 800.0;
index.t = 800.0 - index.t;
// Do a lookup into the environment map
vec3 envColor = vec3 (texture2DRect(whichTexture, index));
// Add lighting to base colour and mix
vec3 base = LightIntensity * BaseColor;
envColor = mix(envColor, base, MixRatio);
sum = vec4(envColor, 1.0);
}
Application code:
void renderScene(void)
{
...
// draw lower right image - Shi
if (mode == 's')
{
if (currentWindow == 1)
// draw environment mapping - Shi
{
gluLookAt(0.0,3.0,6.0,
0.0,0.0,-1.0,
0.0f,1.0f,0.0f);
effectLoc = glGetUniformLocation(p, "whichEffect");
glUniform1i(effectLoc, 6);
//glRotatef(rotAmount, 0, 1, 0);
glRotatef(mouseWindowX*90+90,0,1,0);
glRotatef(mouseWindowY*90,0,0,1);
glutSolidTeapot(1);
}
...
}
...
glutSwapBuffers();
}
- Memo:
Reference: course note.
This effect can only be seen in fullscreen mode; it doesn't display properly when in overview mode.
4.6 Spring Bump Festival
- Overview:
Second page, upper right.
It's a bump mapping effect.
- Principle:
The goal is to draw 8 columns of lanterns, each containing 16 of them.
Calculate new normals for each pixel procedurally using pre-defined math functions, then apply lighting using computed normals value.
- Code:
Vertex shader:
uniform vec3 LightPosition;
varying vec3 EyeDir;
varying vec3 LightDir;
attribute vec3 Tangent;
if (whichEffect == 7)
// bump mapping
{
EyeDir = vec3(gl_ModelViewMatrix * gl_Vertex);
// compute normal and tangent and binormal
vec3 n = normalize(gl_NormalMatrix * gl_Normal);
vec3 t = normalize(gl_NormalMatrix * Tangent);
vec3 b = cross(n, t);
// convert light direction from eye space to tangent space
vec3 v;
v.x = dot(LightPosition, t);
v.y = dot(LightPosition, b);
v.z = dot(LightPosition, n);
LightDir = normalize(v);
// convert eye direction from eye space to tangent space
v.x = dot(EyeDir, t);
v.y = dot(EyeDir, b);
v.z = dot(EyeDir, n);
EyeDir = normalize(v);
}
Fragment shader:
varying vec3 EyeDir;
varying vec3 LightDir;
if (whichEffect == 7)
// bump mapping
{
part = texture2DRect(whichTexture, TexCoord.xy);
// changed the parameters
vec2 BumpDensity = vec2(8.0, 16.0);
float BumpSize = 0.1;
float SpecularFactor = 0.5;
vec3 litColor;
vec2 c = BumpDensity * vec2(TexCoord.st / 800.0); // bump number
vec2 p = fract(c) - vec2 (0.5); // angle of normal changing over the bump area
p.y /= 2.0; //p.y ranges from -0.25 to 0.25 for each bump
float d, f;
d = p.x * p.x + p.y * p.y; // d is the 'radius' of this fragment from center of its bump
f = 1.0 / sqrt(d + 1.0); // f normalizes length of the normal: sqrt(x^2 + y^2 + z^2)
// if we are outside the area of a bump then normal points straight out Z w/ length 1
if (d >= BumpSize)
{
p = vec2(0.0);
f = 1.0;
}
// create a new normal of length 1 for this fragment
vec3 normDelta = vec3 (p.x, p.y, 1.0) * f;
litColor = part.xyz * max(dot(normDelta, LightDir), 0.0);
vec3 reflectDir = reflect(LightDir, normDelta);
float spec = max(dot(EyeDir, reflectDir), 0.0);
spec = pow(spec, 6.0);
spec *= SpecularFactor;
litColor = min(litColor + spec, vec3 (1.0));
sum = vec4 (litColor, 1.0);
}
Main program:
void renderScene(void)
{
...
// draw lower right image - Shi
if (mode == 'w')
{
gluLookAt(0.0,0.0,5.0,
0.0,0.0,-1.0,
0.0f,1.0f,0.0f);
if (currentWindow == 1)
// draw bump mapping - Shi
{
glRotatef(rotAmount, 0, 1, 0);
effectLoc = glGetUniformLocation(p, "whichEffect");
glUniform1i(effectLoc, 3 + 4 * currentWindow);
drawPolygon(-1.5f, -1.5f, 1.5f, 1.5f);
}
...
}
...
rotAmount += 1.5;
glutSwapBuffers();
}
- Memo:
Reference: course note.
I changed some parameters from the example procedural bump mapping code to make the original circles look like Chinese lanterns.
5. What I've been through - The Development Log
First several days: The idea of recording the development log had not come to me at that time, so there is no record for the beginning days. In general, these days I learned GLSL, read some sample code and managed to figure them all out.
1/23 Add: Use keyboard to control grids
1/23 Add: Use mouse to control
1/24 Need: Install OpenCV on my macbook. (This is the day I suddenly noticed that I can't give a presentation in Cyber-Commons using my desktop PC)
1/26 Add: 16*16 mosaic effect
Based on Chris Qin, changed the code according to the characteristic of Texture2DRect()
1/26 Add: sculpture effect
Based on the Chinese website that referred by Chris Qin, changed the code from HLSL to GLSL
1/26 Add: failing old photo effect
Based on Hyejung Hur, couldn't make it beautiful
1/27 Add: updated old photo effect
Discussed with Hyejung Hur, changed some code of old photo effect
1/28 Add: Use keyboard to change pages
1/28 Need: Rewrite renderScene() since environment mapping is using a different method to render the scene.
1/28 Solve: Rewrote renderScene(), still got problem with teapot in grid mode.
1/28 Need: Solve teapot problem in mode z.
1/28 Add: ugly environment mapping effect.
1/28 Add: frame of old photo effect.
1/28 Add: ugly bump mapping effect.
1/28 Need: 1. can't go into bump mapping effect after entered environment mapping; 2. problem with fullscreen bump mapping.
1/28 Solve: solve both of above by realizing that 0 - 800 coordinate system should be used instead of 0 - 1.
1/29 Need: Find setNoiseFrequency() and noise3() to continue writing old photo effect.
1/30 Solve: Got above two functions from Andy.
1/31 Solve: 1. Planted application to Mac OS; 2. can't go into bump mapping problem disappear itself.
2/1 Add: more keyboard functions to let users come back to grid mode whenever current page is changed, so that they won't be confused.
2/1 Add: Changed circles in bump mapping effect to Chinese lantern.
2/2 Add: Gave different window different window titles to give users feedback.
2/2 Add: Passive mouse motion control of environment mapping effect. (Finally figured out the reason it didn't work earlier is that I forgot to add glutPassiveMotionFunc() in main OpenGL application.
2/3 Add: Finally knew why noise effect couldn't work. The reason is I forgot I commented make3DNoiseTexture();in main application.
2/4 Need: 1. screen recording software on Mac OS; 2. stop the application from swallow all the memory.
2/4 Solve: found a screen recording software on Mac OS.
2/5 Need: stop the application's more than 100% CPU occupancy rate.
2/5 Solve: checked whole system, then checked whole code, after making some modification, problem solved.
2/5 Add: a demo video and some snapshots.
6. Who to Thank
Andy Johnson
lighthouse3d.com
Hyejung Hur
Chris Qin
ozone3d.net
Brian Mykietka
Wojciech Niedbala
Epilogue. What I Learned
Looking at the development log, seems to be a easy job. But actually the process is not as straightforward as recorded in the log. One line of "Add" in the log maybe means a whole day of reading, coding, thinking, debugging and frustrating.
Well, but who can say that was not fun experience, playing with and played by the full of craft shaders.
It's a pity that I don't have time to learn more. Because, at the first week, I learned too inefficiently, stuck by a lot of irrelevant material, took too much time to choose effects (Gee there are too many fabulous effect attracting me when I going deeper and deeper in shading world), I have no time to finish the third page. Forgive me, geometry shader.
Even though this time is unprecedented in term of 'how early to start', I think I still have lots of space to improve on calendar planning, especially on multi-threads planning - I didn't realize CS 422 project is so time-consuming.
If I got free time later, I'll come back to make this project more beautiful.
Page updated
Google Sites
Report abuse