Making an efficient tree LOD with impostor baker +!

LOD technic in foliage rendering is widely used in video games. However, a great part of them are just ‘looks right’ in some specific angle of view(usually viewed on the ground). But if the player wants to move to a higher perspective, the trees at a far distance will appear ‘billboard-like’ and become problematic in shadowing. A common way to avoid this(take the Red dead redemption2 for example) is to lay down the billboard a little bit against the view direction and limit the view angle(not close to 0 degrees to zenith angle) at the same time, but evidently that’s not a robust solution at some extent.

Fig 1: LOD transition in UE4 Demo — A boy and his kite. Notice the inconsistency of material and geometry switching with LOD level changes. Things are getting worse when the player tries to look at the trees from a very high perspective due to the using of the billboard.

Fig 2: The main reasons for the inconsistency of LOD switching are sudden changes of texture and geometry.

Here’s some GIF example which I grab from a UE4 official demo — ‘A boy and his kite’ to show how poor the lower tree LOD behaves when the view distance changes rapidly. Notice that both the geometry of tree LODs and the texture changes suddenly, especially on the last LOD switching. Besides of this, the insistence of directional shadow and ambient occlusion also contribute to the problem.

Ryan Bruck has made a UE4 plugin called ‘impostor baker’ to solve the problem by pre-baking all the G-Buffer info in a different angles of view into render textures and can solve the issue of inconsistency in shape and textures fundamentally. But unlucky the tool is not completed very well(only support albedo and normal baking) and comes with lots of glitches and bugs.

ictusbrucks/ImpostorBaker

To make the plugin work well in foliage rendering, I did some minor modifications and the result is quite pleasant in looking.

Fig 3: Final result of using enhanced impostor baker plugin.

Fig 4: Player can hardly notice the difference when LOD changes, even when the trees are very far.

Fig 5: the impostor at the far distance can be rendered correctly as a real geometry.

Here are some modifications I would like the highlight:

First of all, if you are using this plugin in the version after 4.21, a compile error will be run into for sure:

[SM5] Shader attempted to bind the Primitive uniform buffer even though Vertex Factory FLocalVertexFactory computes a PrimitiveId per-instance. This will break auto-instancing. Shaders should use GetPrimitiveData(Parameters.PrimitiveId).Member instead of Primitive.Member.

This is because of the upgrade of the UE4 built-in shader API and the way to solve it is to simply search for all custom nodes in the materials in the plugin and replace ‘Primitive.WorldToLocal’ with:

GetPrimitiveData(Parameters.PrimitiveId).WorldToLocal

As shown in Fig 6, the first thing we need to do is to expand the render target channels to provide enough info for a standard foliage shading model.

The original plugin did this in a very wasteful way by rendering all channels with alpha in separate RTs and this is what we're going to fix.

Fig 6: Add channels to the Map to render in the original plugin(left). The original plugin can only output the G-Buffer channel with alpha individually which is a big waste in memory.

The idea of doing this is to combine depth into the A channel of normal and merge specular, roughness, metallic, and scatter into specular RT. We need to add specular RT into the custom composition BP with a material (Fig 7). If you dive into the plugin BP a little bit, you’ll find that the plugin loops through all RT names from an enumeration list and use the name as an index to find the RT and pass it into the material instance under that name. Since the original plugin material has only exposed albedo and normal as input textures, we also need to also make the specular RT as an input too.

Fig 7: We add one more step in specular map composition. This allows us to pack specular, metallic, roughness, and scatter into a single texture(left). The original plugin simply read every item of the name in the channel enumeration and pass them into the material instance(right).

The packing to specular RT also needs some setups in material binding to create and setup MIDs BP.

Fig 8 : We add a composite material for the composition process in BP. Noticed that the scatter color in G-Buffer is a 3-dimensional vector but we simply pack them into a 1D vector by using a division of BaseColor from ScatterColor. Then we recover the scatter color back in the latter pass.

One more thing to be aware of is to set the Gamma of RT to 2.2 since all texture resources we use are in 8 bits sRGBA format. Without doing this, the packed texture you saved will not be looked the same as original per channel input.

Fig 9: One more thing to be aware of is to set the Gamma of RT to 2.2 since all texture resources we use are in 8 bits sRGBA format.

Now we follow the same steps of the original plugin and save the static mesh and RTs into disk and finally get only 3 textures for a sufficient foliage rendering input.

One more point worth to mention is that the scatter input in G-Buffer is a 3D vector but here I compressed it into a factor of multiplication on the albedo channel(Fig 8) and recover it back for later use.

Fig 10: This will end with a 3 textures input material for a standard tree impostor.

Fig 11: Don’t forget to update the impostor_ThreeFrameBlend material function to connect the packed 4-channels info in specular texture and A channel in Normal texture to G-Buffer output to get a correct result.

The next step is to set up the directional shadow and ambient occlusion to make the shadow behaves consistent when it switches into the impostor.

Because impostor in principle is actually a rotatable billboard with alpha, so the shadow of the directional light is surely not be all right. The way UE4 solve this is to render the geometry and shadow map from both camera point of view(treat light as camera too) and this will end with the result that half of the billboard will be in the shadow. Using screen pixel depth offset can somehow alleviate the effect by offsetting some shadow area into much far distance but the problem still persists.

In this case, we need to get the UE4 distance filed shadow involved to solve the problem. The distance field can keep the occlusion info or an original mesh in the scene thus a correct shadow relationship can be restored as a result.

Due to the limitation of the resolution of distance field texture, we still use a cascaded shadow map(2 layers in this case) at 10000m for a short distant shadowing and distance field shadow for shadowing from 10000m to 120000m as a supplementary.

Fig 12: We tune a DFAO a little bit for a better near AO and use SSAO for a long-distance AO. No glitch will happen in impostor with depth offset in screen space.

In terms of ambient occlusion, the situation becomes opposite since the information on imposter can contribute to G-Buffer, which provides an ideal input of screen space ambient occlusion and therefore we use SSAO as the main method and DFAO as a balancing AO tuner.

Don’t forget to set the effective SSAO distance to the same as distance field shadow to get a unified result.

Fig 13: The balance of CSM and distance field shadow is also quite important. Pay attention that the distance field shadow should be the same as SSAO setting.

Because the geometry shape on the impostor is actually from an alpha masking texture, the last thing we need to worry about is to adjust the alpha clipping to make it looks closer to the geometry in last level.

Fig 15: Just leave an alpha mask offset in the impostor material for better alpha fitting in LOD.

After all of the preparatory work, let’s check out the per-channel output in G-Buffer and move the foliage back and forth to see if any inconsistency happens.

OK, that’s everything prepared.

Comes with some animation shoots of my test result on a big plane.

Fig 16

Fig 17

Fig 18

Fig 19: The debug view of the impostor. Red-dot are all faked trees.

Fig 20: All the true geometries are replaced with the impostor at a higher altitude.