Learning Unreal Engine 4: Implement Cel-Shading w/ Outline Using Custom Shading Model in UE4.22 (1)
Create Your Own Shading Model
Adding a new and customized shading model in UE4 is not something fresh or new. Lots of tutorials can be found, I also learned from many of them, mainly from Matt Hoffman’s Medium: https://medium.com/@lordned
But when I started to follow those tutorials to work on my own version, I found that lots of shader codes are changed in UE4.22, for example, the class “DrawingPolicy” is totally removed. Maybe the entire rendering flow in shaders is not changed too drastically (still base-pass, and then deferred-pass), but some function are removed, separated, or migrated to different files. So you may still need to spend some time to figure them out. So I decide to share this post which is also a note of my implementation in UE4.22.
In addition, I also did some extra effort in my version, like cel-shading for reflection, and which I think is more interesting: doing the outline effect in deferred pass instead of post-processing. When I read those great articles, I found the outline effect is usually done in a final post-processing. So I was kind of curious if I can do the outline effect also in this custom shading model? The answer is yes, and I’ll share my method in this series.
Shading Model or Post-Processing?
In fact, there are many different ways that can achieve cel-shading and outline, such as post-processing, a material graph which might be faster to implement. Shading model is also an alternative one though it may spend more time tracing the UE source code, but I think it also has different advantages that others can’t do. For example, once you finish the shader code of cel-shading model in the deferred lighting pass, it works for point lights and spot lights without extra effort, which is not easily to do with material graph. And also, you can set different parameters for each of the cel-shading material, give them different cel-levels, outline thickness…etc, and a post-processing is difficult to do this.
Material with Your Own Shading Model
First, let’s see how it looks like for a custom shading model:
After adding a shading model, you can select the new shading model in your material editor in the drop-down list. A shading model means a different shading method, for example, PBR is a shading model, Phong shading is another shading model, they use different shading fundamental (or you can say equation/formula). In this article, I’ll also implement the same one as other tutorials do — cel-shading, since it is a relatively simple to do.
When selecting our own custom shading model, we can also define what are the necessary data needed for this shading model? This depends on how you implement your shading. Maybe you need the user to give the input for some threshold, or how many cel bands, or the min/max intensity values to adjust the image contrast. In my practice, I use three values to control my cel-shading and you can see them in the following image.
Cel Bands
This is from the original pin “CustomData 0”. It defines how many “cels” for your cel-shading. For example, if bands = 3, the lighting will be “categorized” into 3 levels. When you have more bands, the shading looks more smooth.
Outline Thickness
This is from the original pin “CustomData 1”. It defines the thickness of you edge for the outline effect. Outline effect is not a necessity for a cel-shading. I just combine them together for fun.
Outline Sensitivity
This is from the original pin “Refraction”. I borrow this pin just because in my practice I won’t support a transparent material with cel-shading so that’s safe to use this pin although it will take you some extra time to modify a little more code of shader compiling pipeline.
Sensitivity is a threshold value checking if a pixel is on edge. There are many algorithms doing edge detection, I just use a very simple one: calculate the difference of the normal vector and depth value and see if they differ large enough.
Modify The cpp / header
To create a new shading model, you will need to modify both cpp codes and shader codes. Let’s see the cpp part first. If you have read Matt Hoffman’s articles, I believe this part would be very easy for you.
1. Adding a ShadingModel Type
In EngineTypes.h, add a new UENUM for the new shading model. For example, MSM_CelShading.
Adding an enum only makes it visible in editor menu. We also need to define the behavior of this shading model. First, we setup the environment for compiling this shader. This is done by adding a “SetDefine([_KEYWORD_])” for this shading model. The SetDefine function means to turn on a #define preprocessor in your shader when compiling. So the following code will make the shader compiler see the “MATERIAL_SHADINGMODEL_CEL_SHADING” as 1
2. Activate The Material Pins
Next, we need to tell the material editor which “pins” can be used for this shading model? (Pins = material attributes that show in your last material node). Please find the function IsPropertyAction_Internal in Material.cpp, there is a switch block testing if the input property (pin) can be active or not.
Let’s take a look at CustomData0 and 1 first, which are “Cel-Bands” and “Outline Thickness” in our new shading model. To make the two pins active, we just add code at the end to test if the ShadingModel is Cel-Shading as well.
3. CustomData
Just to mention some key point here about CustomData, you can leave the name of the CustomData Pin unchanged (so it remains “CustomData0/1”). However, if you want to change the name in editor, you can change it in GetCustomDataPinName() in MaterialGraph.cpp as follow.
Another important thing is that: the range of CustomData 0/1 is between [0.0, 1.0]. Even if you connect a constant or scalar parameter with value > 1.0 to this pin, it will be clamped to 1.0. And that why you see I actually connect the value 1/Bands to the Cel-Bands pin instead of the actual number in my material graph. (And I have to inverse the value in my shader code)
To see why, you can take a look at AllocGBufferTargets() function. UE allocates the GBufferD storing CustomData as B8G8R8A8 format. And CustomData0,1 are used to fill-in the R/G channels as 8-bit. So the value will eventually be clamped to 1.0 as its max value. The code snippet below can be found in SceneRenderTarget.cpp.
4. Borrow The Refraction Pin
You may wonder why I didn’t mention the “Sensitivity” pin? Since this pin is borrowed from “Refraction”, so it is slightly different from CustomData pin. (But still very similar).
First, we do the same thing to make this pin active for Cel-Shading model. As shown below, like we just did for CustomData.
But this is not enough! When I did this and connect some value to the refraction pin, I found it didn’t work. The refraction value in shader is always 0. Why? Because when translating the material graph to shader, UE will check if there are something redundant or not supported, and then the “unsupported” part will be discarded, or be replaced with the default value.
In the Translate() function in HLSLMaterialTranslator.h. The input of refraction pin will be compiled only when the BlendMode is Translucent. So even we make this pin “active” (connectable in editor), the input value is still ignored since the cel-shading is opaque, so it remains 0 which is the default value.
After knowing the reason, it simple to make it work. Just make the refraction pin compiled as well when the MaterialShadingModel is Cel-Shading. It can be done by adding a “||” condition as below.
And there are some extra things you can do (but won’t make anything wrong if you don’t do them). You can find the function RebuildGraph() in MaterialGraph.cpp to change the display name of refraction pin by implementing a GetRefractionPinName() function.
OK, now we’ve finally finished all modifications we need for cpp/header files.
Modify The Shaders: BasePass & GBuffer
The following steps are almost identical with Matt Hoffman’s articles (again). All the difference mainly come from the code migration in UE4.22. If I don’t make something clear enough, you can go back to read his articles :)
1. Setting up a shading model ID
In files ShadingCommon.ush, define a new SHADINGMODELID.
In files BasePassCommon.ush, at the end of this line (which is cut in the following image), add || MATERIAL_SHADINGMODEL_CEL_SHADING) where “MATERIAL_SHADINGMODEL_CEL_SHADING” is what we set in SetDefine in cpp file. Without doing this, the GBuffer will not output the value of our input CustomData.
When BasePass writes output data into GBuffer, it writes the shading model ID as well. Find the ShadingCommon.ush, GetShadingModelID(), add the following line to return the ShadingModelID defined for our cel-shading. (PS, This function is actually called only when RayTracing in 4.22. I’ll explain more later)
2. Setup the ShadingModel Visualization Color
Same as Matt Hoffman’s article again, in ShadingCommon.ush, GetShadingModelColor(), please return some different color (like purple) representing our new shading model, it helps you debugging in editor.
What this does is when you’re in the editor, you can alter the view mode from Lit to Buffer Visualization -> ShadingModel to see if your mesh is rendered with the expected shading model.
3. Ready to Output Data into GBuffer
In BasePass, all we need to do is purely write some information into GBuffer, and no any lighting calculation is done in this pass. (The actual lighting will be calculated in DeferredPass.) In this article, we’ll only focus on what we need to prepare in BasePass.
Up to now, we haven’t really output anything to GBuffer for the deferred pass. The visualization color is only for you to debug in editor.
The main pixel shader of BasePass is defined in BasePassPixelShader.usf. This pixel shader gathers all the required data and renders them all into the GBuffer. Please find the function SetGBufferForShadingModel() in ShadingModelsMaterial.ush, this function handles most of the GBuffer content, including CustomData and ShadingModelID. So in this function, we can put some new code as below: fill-in the CustomData.xy come from the CustomData 0/1 from the material graph.
(And please notice that, in 4.22, the ShadingModelID is actually assigned here instead of what we’ve seen in the previous image: GetShadingModelID(). If you search the function through the whole shader files, you will find this function now is only called from some RayTracing function. The correct place to assign the ShadingModelID in 4.22 would be SetGBufferForShadingModel().)
But in the above image, we only assign the CustomData.xy, which come from the material pins of “Cel Bands”, and “Edge Thickness”. We need to assign the “Sensitivity”, which is from the Refraction Pin, into the GBuffer.
This is done outside the SetGBufferForShadingModel(), it’s just because the value of the refraction pin is obtained by GetMaterialRefraction(), and this function takes different input parameter (PixelMaterialInputs, not MaterialParameters) that is not passed into SetGBufferForShadingModel(), so I did it earlier to avoid changing the function signature.
Next
Up to here, we have already finished the code modification of CPU side, and the shader code of BasePass. You can already create a new material with the new shading model and connect the pins now. And the BasePass will output the correct information to GBuffer now. But we haven’t done any of the lighting calculation part which is the most important part for “shading”.
But I’m gonna make a break at this point since this article is longer than I expected. I’ll put the rest of this series in the next post, including how to add your shading calculation, and a more interesting one, the outline effect, which is shown in the featured image at the beginning. Hope you like it !
