Learning Unreal Engine 4: Implement Cel-Shading w/ Outline Using Custom Shading Model in UE4.22 (2)
Previous Article
This is the second part of this series; the previous article is here.
In last article, we’ve finished the coding work in cpp and the shaders of BasePass, so now it can output some information we need for the lighting. In this article, we’re going to do the core work of implementing the cel-shading. And after reading this, I hope this could be helpful for you to trace the shader code (even just a little :)).
Modify The Shaders: DeferredPass & Lighting
After BasePass rendering, all the necessary information we need for calculating lighting is now gathered in GBuffers. Next, the light renderer will treat each light as a primitive as well (For example, point light is a sphere, spot light is a cone, and directional is a full-screen quad with its light information.) For each light primitive, the deferred shading pixel shader will use the data in GBuffers and the light information to calculate lighting.
1. Lighting Calculation
The entry point of this is the function DeferredLightPixelMain() which is located in DeferredLightPixelShaders.usf. Just like previous UE version, it is still the function GetDynamicLighting() that handles the calculation. In the code snippet below, please remember the place I just inserted the comment, this is where we’ll put the outline effect later.
And keep tracing code into the function GetDynamicLighting() which is in DeferredLightingCommon.ush. This function first gets the shadow term, and then calls the main shading function IntegrateBxDF(). In this function, different shading methods are executed according to the shading model. So let’s add a new shading function in the switch case, and implement a new one call CelShadingBxDF(). You can simply start this function from extending the standard DefaultLitBxDF().
There are different ways to do cel-shading; I just show mine here. Lots of tutorials use smoothstep() or step(). But I would not use them since I’d like to make the Cel-Bands flexible. So that why I add a control value Cel Bands from material pin, and then compute an interval value by discretizing the intensity range [0,1] into several bands. The interval value represents the step value we add when advancing the cell to the next band. The concept is use floor([OriginalVal]x[#Bands]) to get in which band this intensity is, and then multiply by the interval to get the final “rounded” intensity. In this work, I will use the same idea to “cell-ize” all intensity value.
In the following code, you may feel confused with “InvBands” and “Band”. Just recall we’ve mentioned in previous article, the data put in CustomData will be clamped to 1. So this is just a workaround to use the 1/Bands as input, and inverse it back in the shader.
(Actually, I’ve come up with another solution in this article that is use a Global Shader Uniform, or I think it would also work by Material-Parameter-Collection)
The next question is, what is the thing we need to “cell-ize” ? As far as I know, it should be the “intensity” value, that means you should get rid of the color information. For diffuse, it is the value NoL (which means Normal-Dot-Light). This is clearly shown in the above code.
For specular, it could be something related to NoH or VoH, but somehow in UE4.22, the two values are integrated in SpecularGGX() function and it returns the final color directly. To solve this, we can just add an alternative version that returns only the ScaleTerm, and then cell-ize this value, do the color multiplication at the end. The functions SpecularGGX_ScaleTerm() and F_Schlick_ScaleTerm() below show how I do.
OK, after doing this, you can see the basic cel-shading result like this:
2. Apply to Shadow
Actually, we haven’t get the shadowing right. If you follow the steps above, what you see should look like the following image where the shadow is still smooth instead of a cel-shading style.
This is easy to understand. Since we’ve only handled the diffuse and specular intensity, not the shadow yet. And shadow is also a “factor” that scales the lighting. So what to do is to apply the same method to this “shadow factor”.
Please go back to the GetDynamicLighting() function. You can see a function GetShadowTerms() is called before calling IntegrateBxDF(). This function computes the shadow factor and put them in a FShadowTerms “Shadow”. After BxDF, the ShadowTerms is then used to scale the lighting result. So we just apply the same cell-ize processing to this ShadowTerms, which is SurfaceMultiplier in the following image.
Now the shadow attenuation is done as well, back to your editor, recompile shaders, the shadows should be also in the cel-shading style.
3. Apply to Reflection
Beside shadows, we also need to care about the reflections. (Or maybe, if you prefer the reflection keeping its smoothness, you can skip this.)
When you apply the cel-shading model to a glass-like material, with low roughness, the reflection of the sky sphere is shown on the surface. When I did this in my experiment, I just had some weird feeling but I couldn’t figure it out. Not until I applied the material to a sphere did I finally find it was the reflection that makes it look weird: It looks smooth just like the shadows.
For example, from the part inside the red circles, you can see the reflection of the sky sphere is still smooth and therefore makes this glass-like object not so cel-shading-styled.
You can also fix(?) this by same concept but a little more trickier. First, the reflection is done in the function ReflectionEnvironment(), in shader file ReflectionEnvironmentPixelShader.usf. In the reflection calculation, the most difficult thing is to find out “what is able to be cell-ized ?”
Unlike a lighting calculation, the result usually comes from an unlit color and an intensity. So there is no question we just focus on the intensity part. But for the environment reflection, this actually acts more like a “texture fetch”. Not exactly the same but you can think it as pasting some texel to the current pixel. In this case, it is difficult to separate the so called “unlit” and “intensity”. The following is my method. Though it could be not theoretically correct but I think it’s pretty good.
First, the NoV part it the one you can easily do at least, but I found this doesn’t improve much for the cel-shading effect.
My final adopted experiment is to cell-ize the color directly. In the following code snippet, the else {} block is the original code (note the “+=”). In my implementation, I break this operation into a base color (Color.rgb) + an extra color. Honestly I can’t do much about the base Color.rgb, so I focus on the extra part.
My method is to use the LAB color space to replace the missing intensity: Convert the RGB to LAB, and get the “L” value, cell-ize the L value as intensity. And then convert LAB back to RGB. (Note that the range of L is [0,100] instead of [0,1] so you need a range mapping.)
Again, back to editor and recompile shaders, you should see the reflection now shows the cel edges, which makes this glass-like object looks like the cel-shading.
Outline Effect in Custom Shading Model
Finally we are going to the last part of this article. That is how to achieve the outline-effect in a shading model instead of post-processing. This should be an optional effect and maybe isn’t really necessary to implement it in a shading model. Maybe doing this in a post-processing is better and easier, or possible even faster.
1. Idea
But again, this is just for my curiosity to see if we can do something like filtering or fetch data from the neighbor pixels in a shading model? However, the neighbor-fetching behavior is not always correct since you can’t guarantee the neighbor pixels are rendered with their final lit-color, but it is still correct if you only want to use other data than the lit-color, such as depth or normal.
The fundamental idea is that, since we are in a deferred pass, not a base pass, there should be already some valid “whole-scene-information” that is accessible and can be used. That’s right, we can actually access some scene textures. Just like the SceneTextures node in material editor, however, some textures might still be unavailable yet (like SceneColor) since we haven’t really finished the lighting calculation.
2. Add Codes
Back to DeferredLightPixelMain() in DeferredLightPixelShaders.usf which we’ve seen earlier. The better time to do edge detection would be before the lighting calculation, since if a pixel is detected as outline, we can skip the lighting. So let’s insert some code after the line CalcSceneDepth().
The structure SceneTexturesStruct stores several SceneTextures in it, you can get the GBufferA/B/C/DTexture and the scene depth texture here. The normal vector information is stored in GBufferA.
My use a very simple edge detection test:
Sum-up the difference (delta) value from self and the four neighbor pixels, and see if the total difference exceeds the threshold. This is not a best algorithm but since it’s not the key concern of my shading model learning so I’m not going to spend more time on this.
The following is my code with simple comment and it should be easy to understand. Simply fetch the neighbor normal from GBufferATexture, test the normal difference, and then test the depth difference furthermore if needed.
(For your information, actually, the built-in function CalcSceneDepth in UE4 is also fetching the SceneDepthTexture in SceneTexturesStruct. :) )
3. Not Reflection Again
We have the outline now, but if you pay more attention on a glass-like object, you will find the color of the outline seems varies along the surface. No, it’s reflection again!
A very quick way to improve this is to add the same edge detection code in the beginning of the same function ReflectionEnvironment(). Although I think this is kind of tedious and might lead to a poor performance. If possible, I think it would be better to do the edge detection only once and save the result somewhere to re-use.
4. Result
Let’s see the result of what we’ve done up to now! A glass-like object with sky reflection. And a metallic object with some roughness and slight normal mapping.
Final
This is the end of my sharing of doing a custom shading model. As you can see, from the first image below, by implementing cel-shading with a shading model, point lights and spot lights can be supported fairly easily.
In addition, since the outline effect is not done by a post-processing, each mesh can have its own control parameter. In this image, three UFOs have different outline thickness value, which is difficult to achieve by a post-processing.
My sharing of cel-shading model ends here. Hope this is helpful for you. And if you like my sharing, feel free to leave any comments :)
