Stencil shaders

See below for animated version

The stencil buffer allows shaders to use one set of meshes to mask another. For instance, if you want to put a hole in this wall, so you can see if there are any bunnies hiding behind it:

The red circle can be used as a hole

This works by filling in the stencil buffer with a value for those pixels where the hole is. In this case, I am filling it with 1s where the hole is, if you were looking directly forward at the hole:

Stencil Buffer
000000000
000010000
000111000
000010000
000000000

Once those 1s are there, we will do a test in the wall shader to check if there is a 1 at that wall pixel. If there’s a 1, we do not draw that pixel. If there’s a zero, we draw the wall as usual. Essentially it would look like this, if it was written in normal code:

// Wall shader psuedocode
bool drawWallPixel = stencilValue != 1

In actual shader code, here is what the hole shader does, in the SubShader section:

Tags {
   "Queue" = "Geometry-1" // gets to the stencil buffer first
}
   
ColorMask 0   // do not draw this (it's a hole!)
ZWrite off    // the hole has no depth  
   
Stencil {
    Ref 1        // write a 1 into the buffer
    Comp always  // comparison for what is already there.
    Pass replace // Replace anything in the frame buffer with this pixel pass.
}
... Lambert shader ...

Very important point above to be sure that the hole shader is process before any others, by setting the Queue to a value less than the objects that you are wanting to punch out.

Here’s the wall shader:

Tags {
    "Queue" = "Geometry" // needs to be drawn after hole
   }
   
   Stencil {
       Ref 1               // use 1 as the value to check against
       Comp notequal       // If this stencil Ref 1 is not equal to what's in the stencil buffer, then we will keep this pixel that belongs to the Wall
       Pass keep           // If you do find a 1, don't draw it.
   }
... Lambert shader ...

Essentially Ref is the value to test against, Comp is what kind of comparison to make, and Pass is what the resulting action should be.

Full documentation on the values used for stencil buffers

Using multiple stencil shaders and values, like writing a 1 or a 2 depending on which hole is being looked through, you can get trippy effects like this:

From Penny de Byl’s excellent Shader Development from Scratch for Unity with Cg class! More at holistic3d.com