I followed a tutorial that showed part way how Stencil buffer could be used as a viewport.
It confused me when nothing changed when I implemented the tutorial as explicitly stated. Let's say I have a rectangle in OpenGL screen space covering { -1.0f, -1.0f, 0.0f }, { 0.0f, 0.0f, 0.0f }, { 0.0f, -1.0f, 0.0f }, { -1.0f, 0.0f, 0.0f}, or in other words the bottom left quardrant of the entire window. I write to the rectangle only to the Stencil buffer value of '1' and everything else '0' and then draw the object on the screen as normal. Nothing happened!
The reason is because I need to further specify via the glStencilFunc that only stencil value equal to '1' can be drawn.
glEnable(GL_STENCIL_TEST);
glClear(GL_STENCIL_BUFFER_BIT);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);
glStencilFunc(GL_ALWAYS, 1, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glStencilMask(0xFF);
glUniformMatrix4fv(gWVPLocation, 1, GL_FALSE, QMatrix4x4().data());
glDrawElements(GL_TRIANGLES, 6, GL_UNSIGNED_INT, (const void *)(sizeof(GLuint) * 12)); //draw the lower left rectangle in screen space
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
glStencilFunc(GL_EQUAL, 1, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
//draw normally
glDisable(GL_STENCIL_TEST);
Some thoughts of uses: A different pass can write 1 to Stencil buffer of "reflection plane" and in current pass, draw reflection if stencil test for 1 passes. Should improve performance since there exists early stencil test.
VS->Tessellation->GS->early depth, early stencil optimization->FS->normal tests