Extending three.js materials with GLSL

What are three.js materials?

MeshStandardMaterial extended with specular/gloss + instancing +map transformations. See demo.

Material in three.js is a high level abstraction to low level systems both WebGL and Three.js.

Three.js materials all share a common base in order to be able to work with the entire rendering system. All of them share some common properties that work with low level three.js and webgl concepts (including 2d canvas and the DOM), but each has a set of properties or some unique GLSL code that drive some properties that are specific to the material. This article is about the WebGL context.

What do materials do?

Absolutely everything! Almost all rendering operations are somehow tied to a material. The material holds some state that three’s WebGLRenderer uses to set the appropriate WebGL state (is blending turned on for example, or what kind of depth test should be used, if at all…). It also holds some GLSL code (shader), that is used to actually compute things or draw stuff on screen. While Material is the only wrapper around shaders, it’s not the only thing that interfaces with it.

For example, MeshStandardMaterial exposes a roughness parameter to the user. WebGLRenderer maps this to a low level WebGL input type (called uniform). The material also needs the input for the camera, but this is not set through the Material interface, but by calling renderer.render(scene,camera). ‘The renderer then provides this camera input to every material it renders.

Another example are shadows. Once a shadow pass is done, it results with a shadow map (a texture) for a specific light. The materials that are reacting to this light ( .receiveShadow == true ) need this texture as an input. While the user may have made many materials describing them as more or less rough, or more or less blue, the renderer is the only authority on which shadow map should be used, if at all.

It’s a complex system.

What do shaders do?

Computation! Lot’s of it, and in parallel. Shaders are programs written in a shader language (GLSL in case of WebGL) that run at various stages of the rendering pipeline and are executed in parallel on the GPU. WebGL shaders have two stages, vertex and fragment.

Fragment shaders also called pixel shaders, are the programs that decide what color the pixel on your screen should be. This type of program may run some composition math and combine two inputs (like two different videos), or can run some lighting calculation and shade some model to appear as if it were shiny and metal.

Vertex shaders run the math that transforms models from one 3d space to another. When you want to render some 3d model, you have to provide one set of spatial data to the gpu (preferably once), that describes that model. This means that some numbers decribe a shape of a cat, and some other numbers describe a shape of a dog. There can be tens or hundreds of thousands of points, times the dimension, yielding many numbers to be crunched. If we want to rotate the model compared to how a 3d artist modeled it, we would have to apply a transformation to each one of these numbers. This is not suited for real time graphics and GPUs help here by runing these operations in parallel. WebGL first says “describe a dog” and then says “tell me where it is, and where it’s looking at”. In this second stage, we only have to provide webgl with a few numbers (usually 16 in a form of a 4x4 matrix) and transform an arbitrary amount of points with GPU acceleration. The logic for whether some 3d object is represented through an orthographic projection, or a perspective one, happens in the vertex shader.

What are we trying to solve?

Say we have a high level problem “show a 3d car in the browser”. With three.js this is incredibly trivial and high level. If the car is properly stored in some format that describes a scene graph (like glTF, an artist can set up lights and cameras. If the format supports a compatible three.js material (or vice versa), an artist can set how shiny the car body is, or how dull the tires are.

It may be enough to do something like this:

new Loader.load( ‘someModel.someFormat’, model => scene.add(model) )

Pretty simple. A bunch of files load, a scene graph is built, meshes and materials configured and you get a beautiful shiny realistic looking car.

If you want to add shadows, you would do that on your app level. Through a very high level interface. Even though the shadows are a part of the shader system, the input is on the scene graph (Mesh) and WebGLRenderer itself.

model.traverse( obj => {
 obj.receiveShadows = true
 obj.castShadows = true
})

Perfect, the car just got more beautiful. Now you get tasked to add an interactive creative effect to the car. When you hover over the body of the car, it should deform it. Maybe the surface bulbs up towards your cursor, as if it were a really strong magnet or the body was somehow fluid. Or perhaps it’s part of an elaborate transition, when you change the color in some car configurator and click on the body, it sends a ripple out changing the color originating from that point.

The car still needs to look like a car, be shiny, cast shadows. At certain intervals, during transitions, this effect has to run though.

Let’s assume that this additional effect, just like shadowing, Standard lighting computation etc. has to run in the shaders. We want to use all or most of the code that runs MeshStandardMaterial (thousands of lines) but modify only a small portion of it.

The research

Let’s dig into the code a bit. MeshStandardMaterial like other materials, has a core GLSL description in form of a template:

https://github.com/mrdoob/three.js/tree/dev/src/renderers/shaders/ShaderLib
https://github.com/mrdoob/three.js/blob/dev/src/renderers/shaders/ShaderLib/meshphysical_vert.glsl

The syntax found in these files is half GLSL half three.js specific syntax:

#define PHYSICAL                //GLSL
  varying vec3 vViewPosition;   //GLSL
#ifndef FLAT_SHADED             //GLSL
  varying vec3 vNormal;         //GLSL
#endif                          //GLSL
#include <common>               //NOT GLSL
#include <uv_pars_vertex>       //NOT GLSL
#include <uv2_pars_vertex>      //NOT GLSL

In order to make this code maintainable (when all the includes parse it’s thousands of lines of code) these templates reference something called a ShaderChunk. This is a dictionary of reusable GLSL code snippets that lives on the globally accessible dictionary THREE.ShaderChunk. The additional syntax in form of #include <foo> is particular to three.js, WebGLRenderer internally parses this, by looking up a chunk from this dictionary, and appending it onto a string. The resulting string is a valid GLSL program with no additional syntax.

Chunks can be found here:
https://github.com/mrdoob/three.js/tree/dev/src/renderers/shaders/ShaderChunk

And can be as simple as this:
https://github.com/mrdoob/three.js/blob/dev/src/renderers/shaders/ShaderChunk/alphamap_fragment.glsl

#ifdef USE_ALPHAMAP 
  diffuseColor.a *= texture2D( alphaMap, vUv ).g;
#endif

If WebGLRenderer encounters a material that has an alpha map, it will compile the shader in such a way to trigger this branch. Once this branch runs, the order of operations is, texture lookup is done on map alphaMap at the interpolated 2d coordinate vUv , the second component of the resulting vector is selected .g and the diffuseColor's 4th component .a is multiplied by it.

At a high level, each one of these chunks does something to transform the shader inputs, into shader outputs. Some declare the inputs, some write the outputs, some transform intermediary values. Like the alpha map one, which is applied to the final color, or the constant alpha value already present.

The conclusion is, we can modify any material by tweaking it’s GLSL code, as little or as much as we would like.

The problem

How do we modify the material? Let’s consider the naive approach first:

We have forked three.js, and we have access to these shader files. We can modify the code there, but each time we want to see a change we have to compile the entire library. Those .glsl files need to be parsed and “baked” into that THREE.ShaderChunk dictionary, otherwise it won’t run correctly at runtime.

A valid GLSL program does not exist until three.js parses this custom syntax. This, for most apps, happens the first time an mesh with that material is rendered.

This works, but on the downside, it’s tedious, has overhead, and you end up with a different version of three.js.

The copy paste route

The official way of extending a shader can be found in this example:

https://github.com/mrdoob/three.js/blob/dev/examples/webgl_buffergeometry_instancing_lambert.html

We would find the shader template we wish to extend, copy it’s contents over and construct a generic THREE.ShaderMaterial. ShaderMaterial doesn’t have to take valid GLSL, #include statements can be parsed within it as well.

The idea is that you would have an overview in the pasted, unrolled code, and that you could easily tell where to add your own.

This definitely works, but has some drawbacks.

It’s verbose, you actually have to copy over the entire recipe, or resort to string manipulation if you try to reference it. The example above only adds two lines of code, but has to copy over the entire template!

The inputs get obfuscated. ShaderMaterial‘s inputs are on the .uniforms dictionary property, and each on their own is a dictionary (we have to access .value):

var myStandardMaterial = new THREE.MeshStandardMaterial()
myStandardMaterial.roughness = 1 
var myExtendedStandardMaterial = new MyExtendedStandardMaterial()
myStandardMaterial.uniforms.roughness.value = 1 //interface changed

The monkey patch route

Another valid approach, some times suggested is to monkey patch the THREE.ShaderChunk dictionary. Since it is global, and it is public, one can replace any chunk in there.

//before loading your app
THREE.ShaderChunk.some_chunk = my_chunk

Doing this goes against some best practices and has it’s drawbacks but it works.

It would be disastrous if we modified this at runtime, but as soon as three.js mounts, we can do a one time patch which should be safe. Three now globally uses your version of the chunks.

What if we don’t want each instance of MeshStandardMaterial to use the same chunks? Tricky, because we’ve overwritten the global dictionary. We can though, do something like this:

#ifdef MY_DEFINE
  myLogic()
#else 
  
  threeDefaultLogic()
#endif

Piggy back on the preprocessor directives. This way we can have both our modified chunk logic and the default one. We would control this as such:

var myDefaultMaterial = new THREE.MeshStandardMaterial()
var myModifiedMaterial = new THREE.MeshStandardMaterial()
myModifiedMaterial.defines.MY_DEFINE = '' //triggers our branch

This can now work but still has limitations. Some transformations that happen in the middle of the shader can be safely added. Something that depends on a geometry attribute as input (like instancing) can also safely be added, but hooking up uniforms is extremely involved.

The onBeforeCompile route

What if we could tell three.js that when it picks up the template for processing (in order to generate a valid GLSL program), it doesn’t exclusively sample the chunks from that one THREE.ShaderChunk dictionary?

We could provide the material instance with our own chunk, without modifying the global dictionary or using preprocessor tokens.

This is possible with a callback on the Material class called .onBeforeCompile. This is a very powerful and amazing feature, but with a somewhat ambiguous name.

It is invoked by WebGLRenderer before two important steps — parsing and compiling. The valid GLSL program has to be compiled by WebGL, this always happens at runtime. Before this, WebGLRenderer transforms the custom syntax into valid GLSL, this could be called parse time. onBeforeCompile actually happens before this parse time. The argument passed through is a shader object, containing the template syntax. This is why you can use non GLSL #include statements even in ShaderMaterial. At this time, you have the opportunity to intercept this template, and parse the #include statement yourself. Once this happens, three.js will still parse the remaining statements in a discrete step, before actually compiling the shader.

Let’s take that alphamap_fragment chunk for example and modify it.

#ifdef USE_ALPHAMAP 
  diffuseColor.a *= texture2D( alphaMap, vUv * 2. ).g;
#endif

For every diffuse or specular map, we’re going to tile the alpha map twice in both UV directions. Other maps use vUv , we will tell this one to use vUv * 2.

To apply this to some instance of some Material :

var myMaterial = new THREE.MeshStandardMaterial()
myMaterial.onBeforeCompile = shader => {
  shader.fragmentShader = //this is the fragment program string in the template format 
    shader.fragmentShader.replace( //we have to transform the string
      '#include <alphamap_fragment>', //we will swap out this chunk
      require('my_alphamap_fragment.glsl') //with our own
    )
}

Now, when the material is parsed, before three gets a chance to swap out that #include statement with a corresponding value from the THREE.ShaderChunk dictionary, we swap it out ourselves. Three’s parser then simply doesn’t find it.

Let’s swap out the hardcoded value 2 with a dynamic input.

First we tell the chunk to use myValue instead of 2.

#ifdef USE_ALPHAMAP 
  diffuseColor.a *= texture2D( alphaMap, vUv * myValue ).g;
#endif

Then we inject that input into the shader:

var myMaterial = new THREE.MeshStandardMaterial()
myMaterial.userData.myValue = { value: 2 } //this will be our input, the system will just reference it
myMaterial.onBeforeCompile = shader => {
  shader.uniforms.myValue = myMaterial.userData.myValue //pass this input by reference
   
  //prepend the input to the shader
  shader.fragmentShader = 'uniform vec2 myValue;\n' + shader.fragmentShader
  //the rest is the same
  shader.fragmentShader = 
    shader.fragmentShader.replace(
      '#include <alphamap_fragment>', 
      require('my_alphamap_fragment.glsl') 
    )
}

Whenever we change the myMaterial.userData.myValue.value the shader will update. Now we can tile the alphamap independent of other maps.

While powerful, this approach also has drawbacks. It’s impossible to tell what’s in the onBeforeCompile function if you receive an instance of a Material that has already been extended.

It’s verbose. For the most part, we would be doing the same thing over and over again — given some chunk name, replace it with some string. Three.js could do this internally, if we could only provide a key with a string value, three could use that instead of THREE.ShaderChunk without the need for us to manually parse the template.

However, these onBeforeCompile callbacks could be chained together, as long as we don’t have two extensions colliding by modifying the same chunk, several effects could be stacked together.

Some code

You can get pretty creative with onBeforeCompile. For example, you can parse the entire shader yourself and then look for patterns on a more granular level than just swapping out chunks.

This demo, adds some texture transform properties to each someMap slot of a material, by running some regexes on the entire compiled shader.

pailhead/three.js

Some gotchas

If you want your modified material to work with shadows, and you’ve done some kind of additional transformation of the vertices, you need to use Mesh.customDepthMaterial that has the corresponding extension ie. whatever you apply to some mesh material, you need to apply to this:

myMesh.customDepthMaterial =   new THREE.MeshDepthMaterial() 

onBeforeCompile has some issues with hashing, you can’t get too fancy with the logic in there.

Some thoughts

If it were possible to pass own THREE.ShaderChunk dictionary to any THREE.Material i believe it would be the most flexible solution for working with a chunk system.

When monkey patching, we only have the defines to pass around and clone, all the shader code is global. Instead of having many per material branches in a chunk (ie 5 different #ifdef FOO_MATERIAL ) we would just pass the appropriate snippets to appropriate materials. This way the chunk doesn’t have to know about all the possible materials that could be using it.

Also having the code available as soon as you instantiate materials would allow them to be modified and cloned better. With onBeforeCompile this is a black box, so for example if you download some 3rd party module that uses it, it’s almost impossible to further extend it with your own code that you’re using through onBeforeCompile.

const material = new THREE.MeshBasicMaterial()
material.chunks.begin_normal = myChunk

While it may be possible to store a THREE.ShaderChunk like structure under userData it may be impossible to make a generic onBeforeCompile to use it.

onBeforeCompile has an issue when hashing materials, so if you have something generic like:

const myGenericOnBeforeCompile = shader=>{
  const {customUniforms, customChunks} = this.userData
  Object.keys(customUniforms).forEach(uName=>{
    shader.uniforms[uName] = customUniforms[uName]
  })
  Object.keys(customChunks).forEach(chunkName=>{
   //store `vertex` or `fragment` 
   let shaderStage = customChunks[chunkName].vertexStage 
   shaderStage = `${shaderStage}Shader`
   shader[shaderStage] = shader[shaderStage].replace(
     `#include <${chunkName}>`,
     customChunks[chunkName]
   }
 })
}
myMaterial.onBeforeCompile = myGenericOnBeforeCompile.bind(myMaterial)

As amazing as it seems, it wouldn’t work. No matter what you put in your userData.customChunk the WebGLRenderer would not care, it would see the same material and retreive some instance from the cache (most likely the first one it compiled). It calls onBeforeCompile.toString() for hashing, and all it would ever get is the same generic body :(

I have tried to make a decoupling of the entire material system. Currently in a very rough state but can be seen here.