I originally wrote this for the Best way to mask 2D sprites in XNA? Game Development Stack Exchange question. Anyways in this tutorial I’m am going to go over how to use the Sencil Buffer to mask of portions of and image you want to render using an Alpha Mask. Below is an example of the effect.

The first step is to tell the graphics card we need the stencil buffer. To do this when you create GraphicsDeviceManager we set the PreferredDepthStencilFormat to DepthFormat.Depth24Stencil8 so there is actually a stencil to write to.

graphics = new GraphicsDeviceManager(this) {
 PreferredDepthStencilFormat = DepthFormat.Depth24Stencil8
};

The AlphaTestEffect is used to set the coordinate system and filter the pixels with alpha that pass the alpha test. We are not going to set any filters and set the coordinate system to the view port.<

var m = Matrix.CreateOrthographicOffCenter(0,
 graphics.GraphicsDevice.PresentationParameters.BackBufferWidth,
 graphics.GraphicsDevice.PresentationParameters.BackBufferHeight,
 0, 0, 1
);
var a = new AlphaTestEffect(graphics.GraphicsDevice) {
 Projection = m
};

Next we need to set up two DepthStencilStates. These states dictate when the SpriteBatch renders to the stencil and when the SpriteBatch renders to the BackBuffer. We are primarily interested in two variables StencilFunction and StencilPass.

• StencilFunction dictates when the SpriteBatch will draw individual pixels and when they will be ignored.
• StencilPass dictates when drawn pixels pixels effect the Stencil.

For the first DepthStencilState we set StencilFunction to CompareFunction. This causes the StencilTest to succeed and when the StencilTest the SpriteBatch renders that pixel. StencilPass is set to StencilOperation. Replace meaning that when the StencilTest succeed that pixel will written to the StencilBuffer with the value of the ReferenceStencil.

var s1 = new DepthStencilState {
 StencilEnable = true,
 StencilFunction = CompareFunction.Always,
 StencilPass = StencilOperation.Replace,
 ReferenceStencil = 1,
 DepthBufferEnable = false,
};

In summary the StencilTest always passes, the image is drawn to screen normally, and for pixels drawn to the screen a value of 1 is stored in the StencilBuffer.

The second DepthStencilState is slightly more complicated. This time we want to only draw to the screen when the value in the StencilBuffer is. To achieve this we set the StencilFunction to CompareFunction.LessEqual and the ReferenceStencil to 1. This means that when the value in the stencil buffer is 1 the StencilTest will succeed. Setting StencilPass to StencilOperation. Keep causes the StencilBuffer not to update. This allows us to draw multiple times using the same mask.

var s2 = new DepthStencilState {
 StencilEnable = true,
 StencilFunction = CompareFunction.LessEqual,
 StencilPass = StencilOperation.Keep,
 ReferenceStencil = 1,
 DepthBufferEnable = false,
};

In summary the StencilTest only passes when the StencilBuffer is less than 1 (the alpha pixels from the mask) and does not effect the StencilBuffer.

Now that we have our DepthStencilStates set up. We can actually draw using a mask. Simply draw the mask using the first DepthStencilState. This will effect both the BackBuffer and the StencilBuffer. Now that the stencil buffer has a value of 0 where you mask had transparency and 1 where it contained color we can use StencilBuffer to mask later images.

spriteBatch.Begin(SpriteSortMode.Immediate, null, null, s1, null, a);
spriteBatch.Draw(huh, Vector2.Zero, Color.White); //The mask 
spriteBatch.End();

The second SpriteBatch uses the second DepthStencilStates. No matter what you draw, only the pixels where the StencilBuffer is set to 1 will pass the stencil test and be drawn to the screen.

spriteBatch.Begin(SpriteSortMode.Immediate, null, null, s2, null, a); 
spriteBatch.Draw(color, Vector2.Zero, Color.White); //The background
spriteBatch.End();

Below is the entirety of the code in the Draw method, don’t forget to set PreferredDepthStencilFormat = DepthFormat.Depth24Stencil8 in the game constructor.

GraphicsDevice.Clear(ClearOptions.Target 
 | ClearOptions.Stencil, Color.Transparent, 0, 0);
var m = Matrix.CreateOrthographicOffCenter(0,
 graphics.GraphicsDevice.PresentationParameters.BackBufferWidth,
 graphics.GraphicsDevice.PresentationParameters.BackBufferHeight,
 0, 0, 1
);
var a = new AlphaTestEffect(graphics.GraphicsDevice) {
 Projection = m
};
var s1 = new DepthStencilState {
 StencilEnable = true,
 StencilFunction = CompareFunction.Always,
 StencilPass = StencilOperation.Replace,
 ReferenceStencil = 1,
 DepthBufferEnable = false,
};
var s2 = new DepthStencilState {
 StencilEnable = true,
 StencilFunction = CompareFunction.LessEqual,
 StencilPass = StencilOperation.Keep,
 ReferenceStencil = 1,
 DepthBufferEnable = false,
};
spriteBatch.Begin(SpriteSortMode.Immediate, null, null, s1, null, a);
spriteBatch.Draw(huh, Vector2.Zero, Color.White); //The mask 
spriteBatch.End();
spriteBatch.Begin(SpriteSortMode.Immediate, null, null, s2, null, a); 
spriteBatch.Draw(color, Vector2.Zero, Color.White); //The background
spriteBatch.End();