Stencil Buffer Glows - Part 1

• Download demo - 288.03 KB
• Download source - 99.29 KB

Introduction

As a software engineer, effectively communicating to your end user can be a daunting task. For example, in a CAD package, how do you concisely display which object your user is selecting in a three dimensional scene? In a real-time strategy game, how do you display which units the user is controlling? To tackle this problem, one approach is to outline or glow the model of interest.

Background

Before I delve into the implementation of glows, I feel the necessity to briefly overview the stencil buffer. Specifically, what is it and what does it do?

A stencil buffer is a bit mask on a per-pixel level. In other words, it’s a generalized depth buffer!

To elaborate, consider the logic of the depth buffer in an overly simplified renderer. When I want to render a triangle, I first push the vertices of my triangle into the rendering pipeline. Then, the pipeline transforms my triangle into screen space, rasterizes the continuous triangle data into discrete pixels, and computes my shading equation for each of those pixels. Finally, the renderer executes the depth test. It compares the depth of each of my new pixels to the depth of pixels already in the depth buffer. If the new pixel has a smaller depth, it replaces the old pixel with the new one.

The stencil test operates similarly. However, I can define both the stencil comparison operation and what information the renderer writes into the stencil buffer.

With this in mind, how do I make glows?

Step 1

Render the model into the frame, depth, and stencil buffers.

Step 2

Render a larger scale of the model into the frame and depth buffers while masking pixels against values currently in the stencil buffer. (It might be wise to change your shading equation for this rendering step. For example, if I want a red glow, my shading equation needs to be generating red pixels.)

Sample/Demo Requirements

Since I based this project in DirectX 10, you will need Windows Vista or higher. In addition, I was using the DirectX, February 2010, SDK. You will need to install the February 2010, or more recent, redistributable. Finally, although not a requirement, I would recommend having a DirectX 10 compatible GPU.

Source Code Requirements

First, all the sample requirements also apply to the source code. In addition, you will need to install the entire DirectX SDK, the February 2010 version or more recent. After you install the SDK, include the DirectX and DXUT headers and link their respective libraries in your IDE.
Note: Microsoft does not distribute the DXUT libraries. There will be a Visual Studio project within the SDK folder containing the DXUT headers. Build this project to create the libraries.

Using the Code

First, do not forget to allocate a stencil buffer. I know this sounds trivial; however, DirectX 10 combines the depth and stencil buffers into a single object which makes it easy to overlook. Here’s one way to initialize your buffers.

// Initialize a depth stencil buffere suitable for glows.
D3D10_TEXTURE2D_DESC descDepth;
descDepth.Width = pBufferSurfaceDesc->Width;
descDepth.Height = pBufferSurfaceDesc->Height;
descDepth.MipLevels = 1;
descDepth.ArraySize = 1;

// "S8" reserves eight of the thirty-two bits for stenciling.
descDepth.Format = DXGI_FORMAT_D24_UNORM_S8_UINT;
descDepth.SampleDesc.Count = 1;
descDepth.SampleDesc.Quality = 0;
descDepth.Usage = D3D10_USAGE_DEFAULT;
descDepth.BindFlags = D3D10_BIND_DEPTH_STENCIL;
descDepth.CPUAccessFlags = 0;
descDepth.MiscFlags = 0;
V_RETURN( pd3dDevice->CreateTexture2D( &descDepth, NULL, &g_pDepthStencil ) );

// Set up a view for the depth stencil buffer.
D3D10_DEPTH_STENCIL_VIEW_DESC descDSV;
ZeroMemory( &descDSV, sizeof( D3D10_DEPTH_STENCIL_VIEW_DESC ) );
descDSV.Format = DXGI_FORMAT_D24_UNORM_S8_UINT;
descDSV.ViewDimension = D3D10_DSV_DIMENSION_TEXTURE2D;
descDSV.Texture2D.MipSlice = 0;
	V_RETURN( pd3dDevice->CreateDepthStencilView
		( g_pDepthStencil, &descDSV, &g_pDSView ) );

Next, DirectX 10 controls writing to these buffers through a depth stencil state object. Make sure to set up a couple of these corresponding to each rendering step.

On a side note, DirectX implements a double sided stencil. A stencil buffer shadow effect makes the most use of this feature. So, for the glow effect, just ignore the “BackFacing” pixel operations. However, make sure they don’t interfere and adjust values from the “FrontFacing” operations.

// Create depth stencil state corresponding to step one of the article.
D3D10_DEPTH_STENCIL_DESC dsDesc;
dsDesc.DepthEnable = true;
dsDesc.DepthWriteMask = D3D10_DEPTH_WRITE_MASK_ALL;
dsDesc.DepthFunc = D3D10_COMPARISON_LESS;

// Stencil test parameters
dsDesc.StencilEnable = true;
dsDesc.StencilReadMask = 0xFF;
dsDesc.StencilWriteMask = 0xFF;

// Stencil operations if pixel is front-facing.

// Keep original value on fail.
dsDesc.FrontFace.StencilFailOp = D3D10_STENCIL_OP_KEEP;      
dsDesc.FrontFace.StencilDepthFailOp = D3D10_STENCIL_OP_KEEP; 
// Write to the stencil on pass.
dsDesc.FrontFace.StencilPassOp = D3D10_STENCIL_OP_INCR_SAT;
dsDesc.FrontFace.StencilFunc = D3D10_COMPARISON_ALWAYS;

// Stencil operations if pixel is back-facing.
// Since we do not care about back-facing pixels, always keep original value.
dsDesc.BackFace.StencilFailOp = D3D10_STENCIL_OP_KEEP;
dsDesc.BackFace.StencilDepthFailOp = D3D10_STENCIL_OP_KEEP;
dsDesc.BackFace.StencilPassOp = D3D10_STENCIL_OP_KEEP;   
dsDesc.BackFace.StencilFunc = D3D10_COMPARISON_NEVER;

// Create the depth stencil state.
V_RETURN( pd3dDevice->CreateDepthStencilState(&dsDesc, &g_pDSState) );

// Create depth stencil state corresponding to step two of the article.
dsDesc.DepthEnable = true;
dsDesc.DepthWriteMask = D3D10_DEPTH_WRITE_MASK_ALL;
dsDesc.DepthFunc = D3D10_COMPARISON_LESS;

dsDesc.StencilEnable = true;
dsDesc.StencilReadMask = 0xFF;
dsDesc.StencilWriteMask = 0xFF;

// It does not matter what we write since we are not using the values after this step.
// In other words, we are only using the values to mask pixels.
dsDesc.FrontFace.StencilFailOp = D3D10_STENCIL_OP_KEEP;
dsDesc.FrontFace.StencilDepthFailOp = D3D10_STENCIL_OP_KEEP;
dsDesc.FrontFace.StencilPassOp = D3D10_STENCIL_OP_KEEP;
// The stencil test passes if the passed parameter is equal to value in the buffer.
dsDesc.FrontFace.StencilFunc = D3D10_COMPARISON_EQUAL;

// Again, we do not care about back-facing pixels.
dsDesc.BackFace.StencilFailOp = D3D10_STENCIL_OP_KEEP;       
dsDesc.BackFace.StencilDepthFailOp = D3D10_STENCIL_OP_KEEP;
dsDesc.BackFace.StencilPassOp = D3D10_STENCIL_OP_KEEP; 
dsDesc.BackFace.StencilFunc = D3D10_COMPARISON_NEVER;

// Create the depth stencil state.
V_RETURN( pd3dDevice->CreateDepthStencilState(&dsDesc, &g_pDSStateOutline) );

Finally, put everything together and start rendering some glows.

// Set the stencil state corresponding to step one of the article.
pd3dDevice->OMSetDepthStencilState( g_pDSState, 0 );

// This technique renders step one.
g_pOutlineTechnique->GetPassByIndex( 0 )->Apply( 0 );
pd3dDevice->DrawIndexed( 36, 0, 0 );
	
// Set the stencil state corresponding to step two of the article.
pd3dDevice->OMSetDepthStencilState( g_pDSStateOutline, 0 );

// This technique renders step two.
// Remember, this pass needs to do some extra scaling and to output
// the correct color pixels for the glow.
g_pOutlineTechnique->GetPassByIndex( 1 )->Apply( 0 );
pd3dDevice->DrawIndexed( 36, 0, 0 );	

Conclusion

While this implementation produces an excellent result for a cube or sphere, let's consider a torus. When scaled, pixels along the inner ring will fall inside the original, un-scaled torus. This will cause an interesting effect, but not quite a full glow.

In addition, any mesh, which does not uniformly scale around the origin, will have similar visual artifacts.

So, in the second article, I will examine a more robust method for generating the location of the glow. For now, however, concentrate on understanding the stencil buffer. Specifically, how we leveraged it to cull pixels in the second rendering step.

History

• Current revision: Version 1.0