Going by Three.js examples, a common pattern for loading a geometry and corresponding textures looks something like this:

const material = new THREE.MeshStandardMaterial({
  map: new THREE.TextureLoader().load('map.jpg'),
  normalMap: new THREE.TextureLoader().load('normalMap.jpg')
});

const loader = new THREE.JSONLoader();

loader.load('geometry.json', geometry => {
  const mesh = new THREE.Mesh(geometry, material);
  
  scene.add(mesh);
});

TextureLoader.load() returns a Texture, which is updated when the image is loaded. JSONLoader.load() is passed a onComplete callback, which is called when the JSON is loaded and processed. When the callback is called, a Mesh is created, whether the textures are loaded or not.

Sometimes this behavior is desired; you’re showing something as soon as possible. But there are also some issues, which I’m sure you’ve all seen before. If the geometry is loaded before the textures, the model appears untextured, naked, before the textures pop in one by one. As they do, there is a noticeable frame drop while they are processed and uploaded to the GPU.

Fortunately, these issues are easy to fix using Promises.

First, let’s make a wrapper function around a JSONLoader and a Promise.

function loadJSON(url) {
  return new Promise(resolve => {
    new THREE.JSONLoader.load(url, resolve);
  });
}

This function returns a Promise that resolves with the loaded geometry when the file is loaded.

Next we can make a similar wrapper around a TextureLoader.

function loadJSON(url) {
  return new Promise(resolve => {
    new THREE.TextureLoader().load(url, resolve);
  });
}

Note that the TextureLoader also has an onComplete callback like the TextureLoader, though it’s used less frequently. In fact, I’m pretty sure all Three.js loaders conform to this convention.

Now let’s write another wrapper around the texture promises, which will resolve with a Material instance.

function loadMaterial() {
  const textures = {
    map: 'map.jpg',
    normalMap: 'normalMap.jpg'
  };
  
  const params = {};
  
  const promises = Object.keys(textures).map(key => {
    return loadTexture(textures[key]).then(texture => {
      params[key] = texture;  
    });
  });
  
  return Promise.all(promises).then(() => {
    return new THREE.MeshStandardMaterial(params);
  });
}

Promise.all() takes an array of promises and returns a Promise that resolves once all of the sub-promises are resolved.

Next we can combine the geometry and material promise wrappers into one final majestic wrapper function, once again relying on Promise.all().

function loadMesh() {
  const promises = [
    loadGeometry(),
    loadMaterial()
  ];
  
  return Promise.all(promises).then(result => {
    return new THREE.Mesh(result[0], result[1]);
  });
}

Promise.all() resolves with an array of values in the same order as the promises (or values) you supplied. In this case the result array looks like [geometry, material].

With the code above, we have essentially created a mini-loading manager that creates a Mesh once all of its assets have been loaded, all with very little (native) JavaScript. Neat.

Let’s flesh it out a little more. The code below is only one of many possible ways of going about this.

const model = {
  geometry: {
    url: 'geometry.json'
  },
  material: {
    map: 'map.jpg',
    normalMap: 'normalMap.jpg',
    metalness: 0.0,
    roughness: 1.0
  }
};

loadMesh(model).then(mesh => {
  scene.add(mesh);
});

function loadMesh(model) {
  const promises = [
    loadGeometry(model.geometry),
    loadMaterial(model.material)
  ];
  
  return Promise.all(promises).then(result => {
    return new THREE.Mesh(result[0], result[1]);
  });
}

function loadGeometry(model) {
  return new Promise(resolve => {
    new THREE.JSONLoader().load(model.url, resolve);
  });
}

const textureKeys = ['map', 'normalMap']; // etc...

function loadMaterial(model) {
  const params = {};
  const promises = Object.keys(model).map(key => {
    // load textures for supported keys
    if (textureKeys.indexOf(key) !== -1) {
      return loadTexture(model[key]).then(texture => {
        params[key] = texture;
      });
    // just copy the value otherwise  
    } else {
      params[key] = model[key];
    }
  });
  
  return Promise.all(promises).then(() => {
    return new THREE.MeshStandardMaterial(params);
  });
}

The model describes the assets and any additional settings for the Mesh. This is a useful abstraction. We can create many models, and use the code above to create the corresponding meshes. If you need to manage many models at the same time, the createMesh function can be put into another Promise.all() easily. It’s promises all the way down.

const promises = models.map(model => {
  return loadMesh(model).then(mesh => {
    scene.add(mesh);
  });
});

Promise.all(promises).then(() => {
  // load complete! begin rendering
});

Another upside of this approach is that promises can be resolved with with exiting or cached values without any changes to our API. This makes it easy to mix loaded and generated assets.

const model = {
  geometry: {
    geometry: new THREE.SphereGeometry()
  },
  material: {...}
};

...

function loadGeometry(model) {
  if (model.geometry) {
    return Promise.resolve(model.geometry);
  }
  
  if (model.url) {
    return new Promise(resolve => {
      new JSONLoader().load(model.url, resolve);
    });  
  }
}

I’ve written a handful of loading managers over the years, and I can tell you that promises simplify things greatly. You will still need to add logic of your own (caching, error handling, different loaders, additional parameters, etc), but not having to deal with concurrent asynchronous requests yourself is a huge time saver.

The snippets from this post can be found here. Snippet 06 contains all the final functions, and should work if you paste it.

I used a similar approach as outlined in this post in https://nike-react.com/, which was by far the most asset management heavy project I’ve ever worked on. Promises, especially Promise.all(), were instrumental in keeping things manageable.

Promise loading with Three.js was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

I’m a big fan of everything fuzzy and furry, and for a while now I’ve been thinking about a way to implement some of that goodness in WebGL. Recently, as part of a series of code experiments at DPDK, I finally found an excuse to dig into this.

In this post, I will walk though my process and the (mis)steps I took to get to a result that worked out surprisingly well.

Before I got started, I had the following outline in mind:

• Take a geometry (any geometry would do, but I wanted to start with the DPDK 3D plus model we’ve been using for our other experiments).
• For each vertex of the geometry, create a “hair”. This would be a thin and tall THREE.ConeGeometry. The geometry would have multiple height segments to support bending.
• Give each hair a default growth direction. This could be based on the vertex normal.
• Apply a force (gravity) to the hair geometries so they would curve downward.

I didn’t want to go down the simulation route. This would involve physics and more math than I can handle. Particularly hairy geometries would also likely be too intensive to compute in real time. Instead, I wanted to go for a convincing fake, using Three.Bas to manipulate the hair geometries with maximum performance.

At first, I wanted to use data textures in the vertex shader to calculate and store positions for each vertex of the hair geometry. However, this would involve additional render passes, and it felt a little too complicated for my use case.

After putting some more thought into it, I decided to try using quaternions to rotate each hair vertex. The amount of rotation would depend on the vertex y position (between 0.0 and the length of the hair). This way vertices at the base of the hair would be affected less than vertices at the end, resulting in a nice curve.

The first result of this approach can be seen below.

Implementing Version 1

The Three.js Quaternion class has an unassuming but very powerful method:

Quaternion.setFromUnitVectors(from, to);

This method creates a quaternion needed to transform a unit vector from (often representing a direction) to another unit vector to.

All the vertices in a Cone geometry are spread around the Y axis, so the default “direction” for the cone is up, or vec3(0.0, 1.0, 0.0). To achieve the curvature I was looking for, I added two quaternion attributes to the geometry:

• rMin, representing the rotation from UP to the neutral hair direction, based on the vertex normal.
• rMax, representing the rotation from UP to the hair direction + gravity. Since gravity points downward, this boils down to a vector with a lower y value.

Below is a simplified implementation of this. If you are unfamiliar with Three.bas and its API, please refer to my previous tutorial series.

geometry.createAttribute('rMin', 4, (data, i) => {
  const direction = directions[i];

  quat.setFromUnitVectors(UP, direction);
  quat.toArray(data);
});

geometry.createAttribute('rMax', 4, (data, i) => {
  const direction = directions[i];
  
  direction.y -= GRAVITY; 
  direction.normalize();
    
  quat.setFromUnitVectors(UP, direction);
  quat.toArray(data);
});

Inside the vertex shader, I use these quaternions to rotate each vertex. The amount of rotation depends on position.y / hairLength. Vertices at the base of the hair are rotated to the neutral rotation. Vertices towards the end are rotated more downward, because gravity affects them more.

float frc = position.y / hairLength;
vec3 pMin = rotateVector(rMin, position);
vec3 pMax = rotateVector(rMax, position);

transformed = mix(pMin, pMax, frc);
transformed += hairPosition;

The hairPosition at the end is simply the position of the hair on the underlying model. It’s the same for each vertex in a single hair.

With this wired up and working, I added the growth animation logic along with some randomness throughout for the fuzzy look. Below you can see a version of the pen without these frills.

Once more, but with forces

Next I wanted to implement some sort of faux physics logic so the hair could actually react to the mesh being moved around. This is where things get more complicated.

Freeing up gravity

In the version above, the rotation change created by the gravity is calculated on the CPU and stored in an attribute. We do not want to update attribute buffers each frame (this is incredibly slow), so the gravitational force is essentially stuck, baked into the attribute data. To get it unstuck (without changing the approach entirely), I decided to move the quaternion calculation logic to GLSL.

The first step was to port the Three.js Quaternion.setFromUnitVectors method from JavaScript.

vec4 quatFromUnitVectors(vec3 from, vec3 to) {
  vec3 v;
  float r = dot(from, to) + 1.0;
  
  if (r < 0.00001) {
    r = 0.0;
    
    if (abs(from.x) > abs(from.z)) {
      v.x = -from.y;
      v.y = from.x;
      v.z = 0.0;
    }
    else {
      v.x = 0.0;
      v.y = -from.z;
      v.z = from.y;
    }
  }
  else {
    v = cross(from, to);
  }
  
  return normalize(vec4(v.xyz, r));
}

As you can see, the method itself isn’t that complicated. There are more conditional steps than I’d like to see in GLSL, but I don’t dare remove them.

With this logic available on the GPU, I moved the gravity into a uniform, so it could be used in the shader.

uniform vec3 gravity;
attribute vec3 baseDirection;

...

float frc = position.y / hairLength;
vec3 to = normalize(baseDirection + gravity * frc);
vec4 quat = quatFromUnitVectors(UP, to);

transformed = rotateVector(quat, transformed);

In the code above, baseDirection is the neutral hair direction (the one based on the vertex normal). Gravity is stored as 3D vector (with a negative y value). Adding a fraction of the gravity to the baseDirection yields a new direction. Then we can use the quatFromUnitVectors method to rotate the vertex (which is still facing UP) to match the desired direction.

At this point, the fuzzy mesh looks pretty much the same as before (though I am now using the actual normals in stead of normalized positions like before).

With the quaternion calculations on the GPU, it was time to start figuring out how to apply additional forces based on movement and rotation.

Movement

Since the gravity was already stored in a 3D vector, I simply added a movement-based directional force to it. I renamed gravity to globalForce to reflect this change. To determine how much the mash moved, I stored a previous position alongside the current position, and used the difference between the two.

FuzzyMesh.prototype.setPosition = function(p) {
  this.previousPosition.copy(this.position);
  this.position.copy(p);
  
  this.material.uniforms.globalForce.value
    .subVectors(this.previousPosition, this.position);

  this.material.uniforms.globalForce.value.y -= this.gravity;
}

While this worked, the resulting hair movement was quite sluggish, sometimes a little choppy.

I tried various ways to smooth this out, even going so far as to use short TweenMax animations to change the values. Finally I settled on creating an update function, which would be called each frame regardless of animation. There, I add a fraction of the movement delta to a movement force vector. This movement force vector is scaled down each frame, eventually reaching close to 0 if no new forces are applied.

const movementDecay = 0.7;
const movementForceFactor = 0.5;

this.movementForce.multiplyScalar(movementDecay);
this.movementForce.x += this.positionDelta.x * movementForceFactor;
this.movementForce.y += this.positionDelta.y * movementForceFactor;
this.movementForce.z += this.positionDelta.z * movementForceFactor;

this.material.uniforms.globalForce.value.set(
  this.movementForce.x,
  this.movementForce.y - this.config.gravity,
  this.movementForce.z
);

Note that the values for movementDecay and movementForceFactor are completely arbitrary (though the decay should be less than 1.0). As you can see below, the movement is now much smoother. Nice.

Rotation

As for rotation, my goal was to apply a centrifugal force to the hair. I imagine the actual math for this can be pretty complicated. But it should be easy enough to approximate for this use case, right?

I broke the rotation down into an axis and an angle, which together form the quaternion for the mesh itself (remember that all Three.js objects have both a Euler and a Quaternion describing their rotation, which are kept in sync internally).

Having a separate angle makes it easier to measure rotation speed. I did this similarly to movement; storing a previousAngle alongside the current angle, and using the difference between the two to determine speed. This gave me the centrifugalForce, but I still needed a direction to apply this in.

If you spin an object around the y axis, the centrifugal force gets applied on the x and z axes, pushing away from the center of the object, like Homer so elegantly demonstrates below.

If you spin it around the x axis, the force should push away on y and z. For rotation around the z axis, the force should push away on x and y. My initial implementation was as simple as that.

FuzzyMesh.prototype.setRotationAxis = function(axis) {
  switch(axis) {
   case 'x':
    this.rotationAxis.set(1, 0, 0);
    this.centrifugalDirection.set(0, 1, 1);
    break;
   case 'y':
    this.rotationAxis.set(0, 1, 0);
    this.centrifugalDirection.set(1, 0, 1);
    break;
   case 'z':
    this.rotationAxis.set(0, 0, 1);
    this.centrifugalDirection.set(1, 1, 0);
    break;
  }
}

While I could clearly see there was a pattern here, I could not figure out a way to make this work with an arbitrary rotation axis. This led me down a somewhat roundabout search involving perpendicular vectors, normals, binormals, tangents, and planes.

This was a classic case of overcomplicating the problem. It look me longer than I care to admit to realize that I could use the same method I had been using all along: Quaternion.setFromUnitVectors.

FuzzyMesh.prototype.setRotationAxis = function(axis) {
  const ra = this.rotationAxis;
  const cd = this.material.uniforms.centrifugalDirection.value;
  const q = this._quat;

  ra.set(0, 1, 0);
  cd.set(1, 0, 1);

  q.setFromUnitVectors(ra, axis);

  cd.applyQuaternion(q);
  cd.normalize();
  cd.x = Math.abs(cd.x);
  cd.y = Math.abs(cd.y);
  cd.z = Math.abs(cd.z);
  
  ra.copy(axis);
};

In the method above, the rotation axis and centrifugal direction are first reset to their default values (rotation around the y axis). Then I calculate the quaternion to rotate the default rotation axis to the new rotation axis. Then I use this quaternion to rotate the centrifugal direction by the same amount. Finally I make sure the centrifugal direction values are positive (always pushing away from the center). I also normalize the value (since this is a direction), which I had forgotten to do before.

Inside the shader, the rotation force is calculated based on centrifugalDirection and rotation speed (centrifugalForce). I went through several iterations of how to actually apply this force. In the end settled on the code below (though I may still need to change things up).

// start with the global force calculated in JS
vec3 totalForce = globalForce;
// add centrifugal force for this hair / vertex
totalForce += 
  hairPosition * centrifugalDirection * centrifugalForce;

I use hairPosition there because hair away from the center of the mesh should be more strongly affected by rotational force.

There was one more step to take to finalize the rotation logic. If the mesh is rotated, the hair geometries are rotated along with it. To compensate for this, I had to rotate the globalForce in the opposite direction. Luckily, this is easily achieved by using the conjugate of the meshes quaternion. The conjugate of a quaternion is like the inverse of a matrix; it describes the opposite transformation. Luckier still, the Three.js Quaternion class has this method build in.

this.conjugate.copy(this.quaternion).conjugate();
this.material.uniforms.globalForce.value
  .applyQuaternion(this.conjugate);

Adding some noise

There was one last thing bothering me. When the mesh stops moving, the hair stops moving a little while later. This is good, but all of it stops at the same time. That doesn’t look quite right.

To solve this, I decided to use the trusty sine wave to add a little oscillation inside the shader. I based this on a global settleTime uniform and a settleOffset attribute, randomized for each hair.

totalForce *= 
  1.0 - (sin(settleTime + settleOffset) * 0.05 * settleScale);

settleTime is incremented each frame. settleScale is increased when force is applied (either movement or rotation), and slowly scaled back to 0 each frame.

All of these changes lead to the result below. The bouncy mesh animation is controlled by TweenMax, and the hair simply reacts to the change in position and rotation. Sweet.

Too much rotation

I was pretty stoked about the result, and how well it all seemed to work, but as I was experimenting with different value ranges, I soon realized that I had a serious problem. If the forces, especially gravity, applied to the mesh are strong, the hair that stands close to upright gets noticeably deformed and elongated.

No good.

To try to make sense of why this was happening, I eventually took to Illustrator to help visualize the problem. Below you can see an image showing how the quaternion approach rotates each point on the line.

As you can see, the distances between the vertices (the black dots) become longer the more a vertex is displaced, while they should stay the same. After some deep thought, I realized this was happening because I was rotating each vertex around the same origin (0, 0, 0).

First I tried to mitigate this problem by changing how I interpolate the length fractions, which didn’t really solve anything. I also tried simply scaling down the rotated position based on the amount of displacement, but this just lead to different deformities because the origin was still wrong.

Finally, I figured that there was no way around it: I had to calculate the rotations recursively. I was a little reluctant to go down this path, since I thought it would add a lot of overhead. Luckily, it turned out that I had plenty of leeway there.

The actual implementation did take some of trial and error. The result can be seen below.

defines: {
  'HAIR_LENGTH':  (hairLength).toFixed(2),
  'SEGMENT_STEP': (hairLength / heightSegments).toFixed(2),
  'FORCE_STEP':   (1.0 / hairLength).toFixed(2)
}

...

// accumulator for position
vec3 finalPosition;

// get height fraction between 0.0 and 1.0
float f = position.y / HAIR_LENGTH;
// determine target position based on force and height fraction
vec3 to = normalize(baseDirection + totalForce * f);
// calculate quaterion needed to rotate UP to target rotation
vec4 q = quatFromUnitVectors(UP, to);
// only apply this rotation to position x and z
// position y will be calculated in the loop below
vec3 v = vec3(position.x, 0.0, position.z);

finalPosition += rotateVector(q, v);

// recursively calculate rotations using the same approach as above
for (float i = 0.0; i < HAIR_LENGTH; i += SEGMENT_STEP) {
  if (position.y <= i) break;

  float f = i * FORCE_STEP;
  vec3 to = normalize(baseDirection + totalForce * f);
  vec4 q = quatFromUnitVectors(UP, to);
  // apply this rotation to a 'segment'
  vec3 v = vec3(0.0, SEGMENT_STEP, 0.0);
  // all segments are added to the final position
  finalPosition += rotateVector(q, v);
}

Instead of rotating the vertex around the origin, the loop rotates smaller vectors, equal to the length of the hair divided by the number of height segments, and adds the resulting vectors to the final position of the vertex. The loops breaks when the y coordinate of the vertex is reached. I had to rely on defines here, since GLSL does not support non-static loops.

While this computation is obviously more expensive, the performance didn’t seem to suffer too much.

Final Thoughts

The code for this demo can be found here. It’s still a work in progress, but you should be able to take src/FuzzyMesh.js for a spin.

Currently there are a lot of magic numbers that define how much the various forces impact the hair, and you may need to tweak their values depending on your animation. There may also be ways to simplify some of the rotation math. If you have any suggestions, I’m all ears!

I will continue playing around with this, so expect to see more fuzzy (or perhaps leafy?) WebGL meshes in the near future.

This is the fourth and final in a series of articles about advanced WebGL animation using Three.js and Three.bas, my extension for complex and highly performant animation systems.

In the previous post we created a utilitarian animation to demonstrate how Three.bas operates. In this post we will continue down the rabbit hole and explore how we can use vertex shaders to create more pleasing animation systems.

Easing

First, lets take a look at easing. This is a topic most of you should at least be casually familiar with. The standard easing functions, first formalized by Robert Penner, have made their way into practically every animation tool and library. Simply put, easing functions change the curve of how a value changes over time. These functions also integrate smoothly with GLSL.

The simplest easing function is linear, or no easing at all.

function easeLinear(t, b, c, d) {
  return b + (t / d) * c;
}

The four arguments are as follows:

• time: The current time in the animation (relative to duration).
• begin value: The initial value of the property being animated.
• change in value: The delta of the property being animated.
• duration: The total duration of the animation (relative to time).

These four arguments are the same for all other easing functions. Using a different easing function will change the animation curve while keeping the same signature, making them easy to swap in and out.

Below you can see the quad-in ease function and its GLSL equivalent.

// js
function easeQuadIn(t, b, c, d) {
  return b + (t / d) * (t / d) * c;
}

// glsl
float easeQuadIn(float t, float b, float c, float d) {
  return b + (t / d) * (t / d) * c;
)

Other than the differences in syntax, the GLSL function is just as simple and compact as the JavaScript version.

Easing functions can be simplified further if both the change in value and the animation duration are in the range of 0.0 to 1.0 (the value changes from 0.0 to 1.0 over 1 second).

// original quad-in formula
return b + (t / d) * (t / d) * c;

// begin = 0.0, change = 1.0
return 0.0 + (t / d) * ( t / d) * 1.0;

// 0.0 and 1.0 can be removed safely
return (t / d) * (t / d);

// d = 1.0
return (t / 1.0) * (t / 1.0);

// 1.0 can be removed safely
return t * t;

The same process can be applied to all other easing functions, giving us functions that take a single argument (time) and return an eased value. In the previous post, we determined the animation state based on a progress value between 0.0 and 1.0. This will match up nicely.

Three.bas comes with GLSL implementations of all the standard Penner easing functions in both the four and the single argument variants. It also comes with a Bézier curve based easing implementation similar to CSS.

Now let’s add some easing to the cube animation we created in the previous post. This is easy; all we have to do is add a few lines in our material definition.

material = new THREE.BAS.PhongAnimationMaterial({
  vertexFunctions: [
    // add the easeCubicInOut function
    THREE.BAS.ShaderChunk['ease_cubic_in_out']   
  ],
  vertexPosition: [
    // calculate progress between 0.0 and 1.0
    'float p = clamp(time - startTime, 0.0, d) / d;',
    
    // apply easing function  
    'progress = easeCubicInOut(progress);',

    // ... rest of animation logic
  ]
});

Below you can see the animation with easing applied to it, which makes the motion look more natural. Feel free to switch up the easing functions to see how the motion is effected.

Scale & Rotation

So far we have only animated position, but that’s not the only (affine) transformation we can use.

Scale is pretty easy to implement. All we have to do multiply the vertex position by either a number (for the same scale across all axes) or a vector (for different scale across the axes).

float scale = 0.5 // or calculate with progress and/or attributes

// apply the scale scale to x, y and z
transformed *= scale;

Implementing rotation is a little more involved. The most obvious way may be to use a matrix, and fold translation and scale into it. However, this means storing 16 numbers per vertex. It also removes the ability to calculate position, scale, and rotation independently.

After looking at possible solutions, I settled on using quaternions. A quaternion defines rotation based on a normalized axis (x, y, z) and a rotation around that axis in radians (w). Four numbers in total, which fit neatly into a vec4. The quaternion can then be applied to a vector to rotate it.

// create a quaternion from an axis and angle
vec4 quatFromAxisAngle(vec3 axis, float angle) {
  float halfAngle = angle * 0.5;

  return vec4(axis.xyz * sin(halfAngle), cos(halfAngle));
}

// apply the quaternion (q) to a vector (v)
vec3 rotateVector(vec4 q, vec3 v) {
  return v + 2.0 * cross(q.xyz, cross(q.xyz, v) + q.w * v);
}

Please don’t ask me how or why any of this works. Just think of a quaternion as a kind of pseudo-matrix for rotation alone.

The two functions above are included in Three.bas. They can be used the same way as the easing functions. To animate the rotation, you can calculate the axis and/or angle based on our trusty progress between 0.0 and 1.0.

material = new THREE.BAS.PhongAnimationMaterial({
  vertexFunctions: [
    // add quatFromAxisAngle and rotateVector
    THREE.BAS.ShaderChunk['quaternion_rotation']   
  ],
  vertexParameters: [
   'attribute vec4 rotation;'
  ],
  vertexPosition: [
    // rotation angle based on progress
    'float angle = rotation.w * progress;',

    // quaternion based on axis and current angle     
    'vec4 quat = quatFromAxisAngle(rotation.xyz, angle);'

    // apply quaternion to vector to rotate it  
    'transformed = rotateVector(quat, transformed);'
  ]
});

Let’s add some scale and rotation to our animation. Check out the JavaScript for more details about the implementation.

Interpolations

Up until now have mostly used linear interpolation to animate movement. This works quite well, but it may get boring after a while. Fortunately there are many, many functions and equations we can use instead.

Bézier Curves

Bézier curves are one step up from linear interpolation. They are ubiquitously used in many types of graphics software and graphics libraries. A Bézier curve is defined by a start point, end point and two control points.

Positions along the curve can be interpolated based on a value between 0.0 and 1.0. The GLSL implementation of this can be seen below.

// p0: start position
// c0: control point 1
// c1: control point 2
// p1: end position
vec3 cubicBezier(vec3 p0, vec3 c0, vec3 c1, vec3 p1, float t) {   
  float tn = 1.0 - t;

  return 
    tn * tn * tn * p0 + 
    3.0 * tn * tn * t * c0 + 
    3.0 * tn * t * t * c1 + 
    t * t * t * p1;
}

This formula may be imposing, but once you define it you can simply use it over and over again. The cubic Bézier is also included in Three.bas. To use it, we have to include it the same way as the functions above. We also need to define two additional vertex attributes for the control points.

material = new THREE.BAS.PhongAnimationMaterial({
  vertexFunctions: [
    // add cubicBezier function
    THREE.BAS.ShaderChunk['cubic_bezier']   
  ],
  vertexParameters: [
   'attribute vec3 controlPoint0;',
   'attribute vec3 controlPoint1;'
  ],
  vertexPosition: [
    // names abbreviated to fit on one line
    // s = startPosition
    // c0 = controlPoint0
    // c1 = controlPoint1
    // e = endPosition
    'transformed += cubicBezier(s, c0, c1, e, progress);'
  ]
});

Below you can see our animation with the Bézier curve used to interpolate position. Check out the JavaScript for the full code. The logic to define the control points is completely arbitrary, so feel free to experiment.

Catmull-Rom Spline

A Catmull-Rom spline defines a smooth curve through four or more points. This is similar to a Bézier curve, but the line actually goes through the points instead of only being pulled towards them. Since we are not limited to just four points, we can create complex and smooth motion paths. This does however make splines more difficult to implement.

We will store the spline as an array of points. We could use an attribute for this, but if the spline is long, storing (and duplicating) the data per vertex will become a bottleneck of its own. Instead I tend to store the spline as a uniform, a master spline if you will. Then I use attributes to store offsets from this spline, giving each prefab its own path to follow. Since we only need 3 components (x, y, z) to store the path itself, I often use the 4th component (w) to scale this offset at each point of the spline. Spline interpolation can be applied in any number of dimensions.

In the next example you can see our brave little cubes following a spline using the approach outlined above. The master spline and the offset scales are shown in dark gray. Please refer to the JavaScript for the full implementation (this setup is a little different from what we had before).

Parametric functions

Another wealth of interpolation options can be found in parametric functions. Parametric functions take a one dimensional input (time or progress) and return a multi-dimensional output (position). A basic example of this is using sine and cosine to calculate a position on circle with a given radius.

uniform float radius;

vec2 sample(float progress) {
  vec2 position;
  float angle = progress * PI * 2.0;

  position.x = cos(angle) * radius;
  position.y = sin(angle) * radius;

  return pos;
}

Using this as a basis you can define an endless variety of curves an knots that once again can be interpolated based on a progress value. In the example below we use sine and cosine to create a spiral on the x and z axis, while using linear interpolation along the y axis.

Again, this is a very basic example, but it should illustrate the principle. Parametric functions can get very complex, creating beautiful shapes and motion paths. Do note however that trigonometric functions (like sine and cosine) tend to be on the expensive side.

I hope that this post, along with the ones before it, have given you a better understanding of the inner workings of WebGL and Three.js, along with the tools to help you harness their power. We (myself included) have only scratched the surface of what is possible here. To all of you who want to venture deeper, I wish good luck. And thanks for reading.

For more examples, documentation, and other tidbits, check out the Three.bas github page or my CodePen profile.

This is the third in a series of articles about advanced WebGL animation using Three.js and Three.bas, my extension for complex and highly performant animation systems.

In the previous post we emulated parts of the 3D graphics pipeline to better understand how some key pieces fit together. In this post we will build on this, and take a closer look at what happens on the hardware level when you tell WebGL to do stuff.

My Three.js extension helps create animations where many objects can be moved (transformed) simultaneously. The key to my approach is moving some logic that is typically done in JavaScript (which is executed on the CPU) to GLSL (which is executed on the GPU). To understand the reasons for this, we will first create an animation system using a more traditional approach.

Many Cubes

The animation in question will be quite straight forward. We will create a number of cubes, and transition them from a start position to an end position over time.

First we define a Geometry and a Material. Then we use these to create a number of Meshes. We also define some custom properties to prepare for the animation logic.

var geometry = new THREE.BoxGeometry(...);
var material = new THREE.MeshPhongMaterial();

for (var x = 0; x < gridLength; x++) {
  for (var y = 0; y < gridLength; y++) {
    for (var z = 0; z < gridLength; z++) {
      var mesh = new THREE.Mesh(geometry, material);

      // define animation properties
      mesh.startPosition = new THREE.Vector3(...);
      mesh.endPosition = new THREE.Vector3(...);
      mesh.duration = 1;
      mesh.startTime = 0;

      scene.add(mesh);
    }
  }
}

There is a handful of ways you could set up the animation logic, but we will try to do it in a way that minimizes overhead (to give the CPU a fighting chance).

The crux of our animation logic is linear interpolation in the form of

currentValue = startValue + (endValue - startValue) * progress

This is a generalized formula to calculate a value between a start and an end value based on a progress between 0.0 and 1.0. It has many applications and its use has become quite ubiquitous.

Each frame, we use linear interpolation to determine the current position of each cube. The progress is calculated based on startTime and duration of each cube, and a global time value. This value controls the state of the animation.

function update() {
  time += 1/60;
  
  for (var i = 0; i < cubes.length; i++) {
    var cube = cubes[i];
    var st = cube.startTime;
    var d = cube.duration;
    var sp = cube.startPosition;
    var ep = cube.endPosition;
    // progress in range of [0.0 to 1.0]
    var p = THREE.Math.clamp(time - st, 0, d) / d;
    
    // lerpVectors performs linear interpolation
    cube.position.lerpVectors(sp, ep, p);
  }
}

Below you can see the running animation. Please take a look at the JavaScript to see the complete code. The buttons can be used to control the number of cubes that are created.

Depending on your device, the animation should run fine with a few hundred cubes. It may even be able to handle a thousand. But if you add many more, you will see a sharp decrease in performance.

Have we hit the limits of what WebGL can do?

Too many cubes?

The short answer is no.

For the long answer, we need to examine what is actually happening on the hardware level.

In the previous post we saw how matrices are used to store the transformation for a mesh in a scene. A transformation matrix consists of 16 numbers. Each mesh in our scene has its own transformation matrix. When the position of a mesh is changed in JavaScript (on the CPU), its matrix needs to be updated and sent to the GPU.

The CPU and the GPU are two separate entities connected by a strict communication protocol. The GPU is optimized to perform computation in parallel, meaning it can do a lot of math at the same time. However, when you send new data to the GPU, these computations are interrupted. The new data needs to be processed before the GPU can resume. The more data you send to the GPU, the bigger the interruption becomes.

When our animation system has too many cubes (each representing 16 numbers), the GPU has to spend more time processing new data than it gets to spend on doing what it loves to do: math.

The key to getting the most out of the GPU is minimizing the quantity of new data that it has to process each frame. While we cannot significantly reduce the amount of data we need to calculate the animation state, we can change how the data is handled between the CPU and the GPU.

More Cubes!

Currently, the bottleneck is the JavaScript position calculation that forces the meshes’ transformation matrix to update each frame. To resolve this bottleneck, we can move the animation update logic from JavaScript to GLSL. Since the CPU and the GPU are separate entities in terms of memory, we also need to store the required data on the GPU.

Custom Attributes

When we created the meshes for the animation, we defined four custom properties: startPostion, endPosition, startTime, and duration. This is the data required for the animation state calculation. Since this data is different for each mesh, the best way to store it on the GPU is using custom attributes in the cube geometry.

As mentioned in the previous post, a 3D geometry consists of a list of vertices. You can think of a vertex as a JavaScript object with a number of properties, called attributes. Position is one such attribute, but like a JavaScript object, a vertex can have any number of attributes.

Thinking in terms of JavaScript, adding additional properties would be simple:

var vertex = {
  position: new THREE.Vector3(...),
  startPosition: new THREE.Vector3(...),
  endPosition: new THREE.Vector3(...),
  startTime: 0.0,
  duration: 1.0
};

Unfortunately, adding vertex attributes is a little more involved. While the object representation of a vertex is easy to reason about, objects are an inefficient way of storing (lots of) data in memory. Instead, vertex attributes are flattened (destructured) and stored in arrays called buffers.

Each vertex attribute has its own buffer. Buffers have a fixed size, which is determined upon creation. The size of the buffer depends on two things: the number of vertices and the size of the attribute. For instance, position is defined by 3D vectors, so the size of each attribute is 3 (x, y, z). A single cube has 8 vertices, so the position buffer length would be 8 × 3 = 24. The buffer itself would look something like this:

[v0.x, v0.y, v0.z, v1.x, v1.y, v1.z, v2.x, v2.y, v2.z, ...etc]

As for our custom properties, the attribute size for startPosition and endPositions is 3. The attribute size for startTime and duration is 1. Later on, Three.bas will help us create the buffers of the correct size.

Buffer Geometry

Before we continue, there is one more important thing we need to consider. If you look at how we created the meshes, you may notice that each mesh uses the same geometry and material, in stead of creating new instances. This way Three.js can optimize its internal render logic, and draw all the cubes at once. Each mesh is essentially a copy of the geometry rendered with a different transformation matrix. It is much faster to render the same geometry a thousand times than it is to render a thousand different geometries once.

We will be adding additional vertex attributes to the geometry, and these values will be different for each mesh. Because of this, Three.js can no longer perform its optimizations, so we will have to do it ourselves. Fortunately Three.js does provide a class that will help us: THREE.BufferGeometry. Buffer Geometry makes it easier to work with vertex attribute buffers directly (in stead of the object representations found in THREE.Geometry).

In Three.Bas, I created a wrapper around this class that helps create buffer geometries where base geometries are repeated a given number of times. I refer to this base geometry as a prefab. This wrapper also helps create and populate buffers as discussed above.

// create the geometry that will be repeated in the buffer geometry
prefab = new THREE.BoxGeometry(cubeSize, cubeSize, cubeSize);

// create the buffer geometry where the prefabs are repeated
geometry = new THREE.BAS.PrefabBufferGeometry(prefab, cubeCount);

// create buffers with the appropriate item size
var startPositionBuffer = 
  geometry.createAttribute('startPosition', 3);

var endPositionBuffer = 
  geometry.createAttribute('endPosition', 3);

var durationBuffer = 
  geometry.createAttribute('duration', 1);

var startTimeBuffer = 
  geometry.createAttribute('startTime', 1);

Creating an Animation Shader

Now that the data required to calculate the state of our animation is available on the GPU, we need to move the update logic there too. We will do this by “extending” a built-in Three.js material with a few lines of GLSL.

Three.js has a number of built-in materials, and the corresponding shaders are represented as strings. Because some logic is shared between these materials, the shader code is broken down into chunks, which are concatenated together into the final shader code.

Three.bas builds on this principle by injecting custom shader chunks in key places inside the original Three.js shader code. This allows you to define arbitrary (animation) logic in GLSL, while still being able to use all of the built-in features of Three.js, like lighting. Pretty cool.

The GLSL equivalent of our JavaScript update logic is as follows:

// define attributes and uniforms

uniform float time;
attribute vec3 startPosition;
attribute vec3 endPosition;
attribute float startTime;
attribute float duration;

// ...then inside the main function

float progress = clamp(time - startTime, 0, duration) / duration;
position += mix(startPosition, endPosition, progress);

The attribute declarations must have the same name and type as the buffers we created. We also use two built-in GLSL methods: clamp and mix. clamp will ensure a value is between a upper and a lower bound. In this case, it will make sure progress is always between 0.0 and duration. Diving this by duration will give us a value between 0.0 and 1.0. mix performs linear interpolation as described earlier this post.

Below we use Three.bas to inject this GLSL code into the built-in THREE.MeshPhongMaterial. We also define time as a uniform to be used in the vertex shader. When put together, the custom buffer geometry and shader material can be used to create a single mesh, which is added to the scene.

material = new THREE.BAS.PhongAnimationMaterial({
  uniforms: {
    time: {value: 0}
  },
  vertexParameters: [
    'uniform float time;',
    
    'attribute vec3 startPosition;',
    'attribute vec3 endPosition;',
    'attribute float startTime;',
    'attribute float duration;',
  ],
  vertexPosition: [
    'float p = clamp(time - startTime, 0.0, duration) / duration;',
    'transformed += mix(startPosition, endPosition, p);'
  ]
});

cubes = new THREE.Mesh(geometry, material);
scene.add(mesh);

Like in the previous update method, time is used to control the state of the animation. We still have to set its value in JavaScript however, which makes our update function look like this:

function update() {
  cubes.material.uniforms.time.value += 1/60;
}

Analyzing the Result

Now let’s bask in the glory of the new version of our animation. Check out the full source code for some more details of how everything fits together.

Depending on your hardware, you should be able to go up as high as a million cubes, but in any case you should be able to see between 100 and 1000 times more cubes move at 60 fps.

Wow.

This approach is so much faster because we went down from 16 × number of cubes worth of numbers to be updated and sent to the GPU each frame down to 1. One number, regardless of how many cubes are on screen.

The update logic itself is now executed once for each vertex, in stead of once per mesh. That doesn’t feel great, but the impact of the duplicate computations is negligible. The GPU is just that good at math.

The boost in performance does come at a cost however.

As you may have noticed, there is a significant delay when you create a large number of cubes. This should be no surprise, as we create and store significantly more data when the cubes are created. We are basically opting to pay our CPU cycle dues up front, in stead of frame by frame installments.

This approach is also less flexible. Since we are operating close to the hardware, there are some memory considerations we need to work around.

When the vertex shader is executed, it only has access to the attributes of that particular vertex. This makes it impossible to implement animation systems where positions depend on each other, like flocking.

Shaders are stateless. They cannot store values between executions. They also cannot modify the values of attributes passed to them. This means that “dynamic” logic where values change incrementally based on various inputs are difficult to implement. This will become much easier in WebGL 2. While it’s still far away, I’m very much looking forward to playing around with transform feedback.

There is an alternate animation approach that works around these limitations using textures in the vertex shader. This is a powerful technique, but it has its own drawbacks. It’s a different beast all together.

This concludes the overview of the structure and reasoning behind Three.bas. The animation we created as proof may not have been particularly spectacular, but in the next post we will kick it up a notch by throwing rotation, scale, and more interesting interpolations into the mix.

Matrices are a crucial part of both 2D and 3D graphics development. If you are unfamiliar with this topic, it can seem quite daunting. But you can go a long way with just a high level understanding of the related concepts and a good math library, which Three.js provides.

There are many tutorials and videos explaining how matrices work, but this post will instead focus on how you can use matrices and vectors in Three.js without understanding any of the internals.

The most common matrix you will be working with is a 4 by 4 matrix called a transformation matrix. Similar to the CSS transform property, the types of transformation that can be represented by this matrix include translation (change in position), rotation and scale.

Think of the transformation matrix as a magic black box. If you put a 3D point (called a vector) into this box and “shake” it, a transformed vector will come out. Let’s look at an example.

Say we have a vector with the value{x: 20, y: 20, z: 0} and a matrix representing a translation of {x: 10, y: 40, z: 0}. Putting the vector into the matrix will result in a transformed vector with the value of {x: 30, y: 60, z: 0}.

If we have more vectors we can apply the same transformation by putting them into the same matrix.

Applying the same transformation to different vectors is a common scenario in graphics development. Vectors often represent shapes: 3D geometries and 2D polygons. Applying a matrix transformation to such a shape will transform it without changing the relative positions of the points. This is called affine transformation.

Representation in Code

The equivalent of putting a vector into a matrix is multiplying the vector by said matrix.

transformedVector = vector * transformationMatrix;

In JavaScript, we cannot use operators (+, -, *,/) on custom classes like matrices and vectors. Instead Three.js provides a robust API for working with these classes. The Three.js equivalent of the multiplication above is as follows:

var vector = new THREE.Vector3(20, 20, 0);
var matrix = new THREE.Matrix4();

matrix.makeTranslation(10, 40, 0);

vector.applyMatrix4(matrix);

Note than the original vector is now the transformed vector because applyMatrix4 modifies the vector directly.

By default, a matrix does not represent any transformation. Applying the default matrix, called an identity matrix, will return the same vector you put in. The matrix.makeTranslation(x, y, z)method is part of the Three.js API. It will make the matrix represent a translation based on the supplied arguments. If you ever need to reset a matrix, you can do so by calling matrix.identity().

The API also provides similar methods for rotation and scale. Scale is fairly straightforward, with a method called matrix.makeScale(x, y, z). Rotation is a little more involved, simply because rotation in 3D can get a little complicated. Three.js provides the following methods for rotation:

matrix.makeRotationX(angle);
matrix.makeRotationY(angle);
matrix.makeRotationZ(angle);
matrix.makeRotationAxis(axis, angle);
matrix.makeRotationFromEuler(euler);
matrix.makeRotationFromQuaternion(quaternion);

Note that all angles in Three.js are measured in radians, which represent rotation in PI, with 360 degrees being equal to 2 PI.

The method matrix.makeRotationAxis takes a THREE.Vector3 representing the axis of rotation, and an angle around that axis. It is a condensed version of the three methods above, and provides a way to represent a rotation around an irregular axis. The two lines below are equivalent:

matrix.makeRotationX(Math.PI);
matrix.makeRotationAxis(new THREE.Vector3(1, 0, 0), Math.PI);

A THREE.Euler represents rotation around the x, y, and z axes. This is the most common way of representing rotation. A THREE.Quaternion is an alternate way of representing rotation, based on an axis and an angle.

Depending on what you need to do, and the data you have available, you may end up working with any of these methods. Three.js also provides methods to easily convert between the different representations of rotation.

Finally, the Three.js api provides a method to create a matrix representing a combination of translation, rotation, and scale: matrix.compose.

var translation = new THREE.Vector3();
var rotation = new THREE.Quaternion();
var scale = new THREE.Vector3();

var matrix = new THREE.Matrix4();

matrix.compose(translation, rotation, scale);

Stacking boxes

One of the great things about boxes is that they can be stacked. Similarly, matrices can be “stacked” to create a representation of a combined transformation.

Moreover, this stack can be “squished” into a single matrix. This allows us to represent more complex transformations while using the same amount of numbers. Neat!

The equivalent of stacking matrices and squishing the stack into a single matrix is multiplication.

combinedMatrix = rotationMatrix * scaleMatrix * translationMatrix;

The Three.js API provides 2 methods for multiplying matrices: matrix.multiply(otherMatrix)and matrix.multiplyMatrices(matrixA, matrixB). The first method multiplies the matrix by another matrix. The second method sets the matrix to the result of matrixA * matrixB.

var rotationMatrix = new THREE.Matrix4().makeRotation(...);
var scaleMatrix = new THREE.Matrix4().makeScale(...);
var translationMatrix = new THREE.Matrix4().makeTranslation(...);

var combinedMatrix = new THREE.Matrix4();

combinedMatrix.multiply(rotationMatrix);
combinedMatrix.multiply(scaleMatrix);
combinedMatrix.multiply(translationMatrix);

Multiplying 3 matrices that represent a rotation, scale and translation will result in the same matrix as calling matrix.compose with equal transform values.

Pre-multiplication and Post-multiplication

When working with numbers, multiplying a list will yield the same result no matter the order of multiplication.

1 * 2 * 3 * 4 = 24
4 * 3 * 2 * 1 = 24

This does not apply to matrices. As such, you always have to pay close attention to the order of multiplication. Consider the example below.

Think of it this way: even though the combined transformation is “squished” into a single matrix, the transformations are still applied in the order they were originally stacked. Cool!

Multiplying your matrix by another matrix is known as post-multiplication. Multiplying another matrix by your matrix is know as pre-multiplication. Note that matrix.multiply(otherMatrix) always post-multiplies, while matrix.multiplyMatrices(matrixA, matrixB) can do both.

A common use case for multiplying matrices is traversing up and down a scene graph. A scene graph is a data structure consisting of nodes with children, where each child is also a node that can have children of its own. Many graphics libraries, including Three.js, use a scene graph. The HTML DOM shares many traits of a scene graph as well.

When you transform a node in a scene graph, all of its children get the same transformation applied to them. Any of these children can have a transformation of their own, which is usually relative to their parent node. If you ever need to determine the transformation of a nested node relative to the root, this can be calculated easily by recursively pre-multiplying parent transformations until you reach the root node.

// start with the transformation for this node
var matrixWorld = new THREE.Matrix4().copy(node.matrix);

// get the first parent
var parent = node.parent;

do {
  // pre-multiply
  matrixWorld.multiplyMatrices(parent.matrix, matrixWorld);
  // get next parent
  parent = parent.parent;
}
while (parent);

In Three.js, the transformation of a child relative to the scene is stored in a property called matrixWorld.

The inverse of a matrix

When working with numbers, you can ‘undo’ a transformation. Consider the following:

4 * 5 = 20;
20 / 5 = 4;

This does not apply to matrices. You cannot divide a vector by a matrix, but you can do something similar by multiplying a vector by the inverse of the same matrix.

transformedVector = originalVector * matrix;

inverseMatrix = matrix.inverse();

untransformedVector = transformedVector * inverseMatrix;

// untransformedVector == originalVector

The inverse of a transformation matrix represents the opposite transformation.

Three.js provides a method to calculate the inverse of a matrix:

var matrix = new THREE.Matrix4();

// apply transformation ...

var inverseMatrix = new THREE.Matrix4();

matrix.getInverse(inverseMatrix);

Among other things, the inverse of a matrix is used when working with a camera object in a 3D scene.

As you can see, matrices are a powerful and versatile tool. They can represent a wide range of simple or complex transformations. Vectors can then be transformed using the same procedure, regardless of what kind of transformation the matrix represents.

There is much more to matrices than covered here, but I hope this post has given you some insight into their power, and how it can be tapped to deal with common scenarios in graphics development.

This post will cover some of the basics of writing shaders in GLSL. For a better look at how shaders work, check out my other post in this series.

Shaders in WebGL are written in a language called OpenGL Shading Language, GLSL for short. As the name suggests, it is part of the OpenGL graphics standard, which WebGL is derived from.

If you are looking for WebGL resources, and your search yields no satisfactory results, consider expanding it to include OpenGL. OpenGL has a bigger scope and feature set, but there is still a lot of overlap.

GLSL Syntax

When writing GLSL the first thing you need to remember is to always end a line with a semicolon. If you forget one, your shader will likely crash, but rarely in a way that clearly points to a missing semicolon. As shaders can get highly complex, debugging a error (for hours) only to find out that it was caused by a missing semicolon can be very frustrating (or so I’ve heard…).

The other major difference from JavaScript is the fact that GLSL is strongly typed. GLSL provides a number of built in “classes” for mathematical objects like vectors and matrices. Trying to operate on classes that have no valid interactions, like a 2D vector and a 3D vector, will throw an error.

Critically, this also applies to numbers. There are two types of numbers in GLSL: integers and floats. Integers are round numbers (1, 2, 3, 4, etc). Floats are numbers with a fractal component (1.125, 2.0, 3.142, etc). You cannot use integers and floats in the same operation. This would be heresy!

2.0 + 2.0; // ok!
2 + 2; // ok!
2.0 + 2; // NOT OK!

Next we will take a look at a basic vertex shader, and dissect it word by word.

attribute vec3 position;

uniform mat4 modelViewMatrix;
uniform mat4 projectionMatrix;

void main() {
  mat4 mvpMatrix = modelViewMatrix * projectionMatrix;

  gl_Position = mvpMatrix * vec4(position, 1.0);
}

Shaders have two main parts. The first part (above the main function) is where we declare our inputs (parameters). The second part (inside the main function) is the entry point of the shader. This is where the math happens. Trying to use parameters you have not declared will throw an error. The order of the parameters is important as well, as you cannot reference a parameter before it has been declared.

The void keyword implies that the main method does not have a return value. Instead we use gl_Position to represent the output of the vertex shader. You must declare a gl_Position inside the main function, which makes sense since this is kinda our purpose here. Failure to do so will throw an error. The equivalent of gl_Position in fragment shaders is gl_FragColor Note that the shader execution does not halt after these keywords are used.

Parameter qualifiers

Declaring a parameter has the following syntax:

{qualifier} {type} {name}

attribute vec3 position;
uniform mat4 modelViewMatrix;

The qualifier specifies the source of the parameter. This basically tells the GPU where to look for the data associated with the parameter. It’s important to correctly declare the qualifier, as having incorrect qualifiers will cause the GPU to look in the wrong place, which would be rude.

In vertex shaders, the following qualifiers are available:

• attribute: This is an attribute of the vertex for which the shader is executed. The data will be different for each execution.
• uniform: This is a uniform value, and will contain the same data for each execution of the shader.
• varying: This is a secondary output of the vertex shader that will be passed to the fragment shader. The fragment shader can then access an interpolated value for each fragment in the face.
• const: This is a “static” value. It’s part of the shader, and cannot be set from outside. This is typically used for values that will never change, like the value of PI. Note that an error will be thrown if you declare a const without assigning it a value.

Fragment shaders have access to all of these, except for attribute. Since the fragment shader cannot access vertex attributes, the varying qualifier is used to pass attribute data along. Note that if you are using uniforms, you must declare them in both the vertex shader and the fragment shader. This also applies to uniforms and constants, since shader steps have their own isolated execution scopes.

For variables inside the main function the qualifier should be omitted. These then behave similarly to variables in JavaScript.

Types

Whenever you declare a variable, you must also specify its type. As mentioned before, GLSL has a number of “classes” to math with.

• Numbers: int, float.
• Vectors: vec2, vec3, vec4.
• Matrices: mat2, mat3, mat4.

It’s important to make sure the data you supply to the shader matches the type you specified. Failure to do so will, you guessed it, throw an error.

All of these types behave the same as numbers do in JavaScript. The main advantage of this is that we can use mathematical operators on them, like below:

mat4 mvpMatrix = modelViewMatrix * projectionMatrix;

Note that WebGL does not support the modulus operator (%).

This makes it much easier to express mathematical operations, which is pretty cool. This power does have limits however: you can only use operators on appropriate types.

gl_Position = mvpMatrix * vec4(position, 1.0);

In this line we convert our position attribute (which is a vec3) to a vec4 in order to multiply it by the mvpMatrix (which is a mat4). We have to do this because you cannot multiply a mat4 and a vec3. Trying to do so will throw an error. As a rule, the size of vectors and matrices being operated on must always be the same.

Vector Constructors

vec4(position, 1.0) is a constructor for a vec4. GLSL constructors like this offer quite a lot of flexibility. In this example, the x, y, and z components of the position vector are destructured, and passed to the new vector. Then the fourth component (w) is set to 1.0. Note that this is a float, not an integer. Writing vec4(position, 1) would throw an error, since you cannot put an integer into a float vector. They just don’t get along.

Below are some other ways a vector can be constructed.

vec4 p = vec4(1.0, 2.0, 3.0, 4.0);
vec4 p = vec4(myVec2, 3.0, 4.0);
vec4 p = vec4(myVec2, myVec2);
vec4 p = vec4(1.0, myVec3);

The same logic applies to other the vector types. Trying to create a vector with an incorrect number of values will throw an error.

Vector Components

Once constructed, the vector components can be accessed the same way as in JavaScript: p.x, p.y, p.z and p.w. Since vectors are also used to represent colors, the corresponding components can likewise be accessed using p.r, p.g, p.b and p.a.

You can also access multiple components at once, which implicitly creates a vector of the appropriate size.

vec4 p = vec4(myVec4.xyz, 1.0);

Components can be accessed in any given order, which is referred to as swizzling. So flexible!

vec4 p = vec4(myVec4.zxy, 1.0);

GLSL Arrays

If you want to get fancy, you can also declare arrays. These arrays are also typed, which means they can only contain members of the same type. The line below creates an array of 6 floats.

float myArray[](1.0, 2.0, 3.0, 4.0, 5.0, 6.0);

Declaring an array you intend to populate from outside the shader looks like this:

uniform float myArray[6];

Members of arrays can be accessed the same way as in JavaScript.

float a = myArray[0]; // a === 1.0

Functions

GLSL also supports functions!

float decimate(float x) {
  return x * 0.9;
}

void main() {
  float x = 1.0;
  
  x = decimate(x);
  
  // x is now 0.9!
}

As you can see, the syntax is fairly similar to JavaScript. The main difference is that functions, like variables, must declare a type. The value you return must then be of the same type as specified. If your function does not return a value, the return type must be specified as void. Any functions you declare should be placed above the main function (otherwise you cannot use them inside the main function).

GLSL functions are scoped similarly to JavaScript. If you declare a variable inside the function, you can only access it inside that function. Note however that functions in GLSL cannot be nested.

Argument Qualifiers

If you want to get really fancy, you can use special qualifiers that change how arguments for a function behave.

void decimate(inout float x) {
  x *= 0.9;
}

Whoa! What’s happening here?

By default, any arguments you pass to a function are passed by value. This essentially means that the value is copied into the argument, and the variable you passed to the function cannot be modified by it.

Using the inout qualifier, the argument is instead passed by reference, which means that the original variable can be changed directly.

void main() {
  float x = 1.0;
  
  decimate(x);
  
  // x is now 0.9!
}

One use case for this is creating functions that “return” multiple values.

void decimateTwo(inout float x, inout float y) {
  x *= 0.9;
  y *= 0.9;
}

Another qualifier you can use is out.

void decimate(in float x, out float y) {
  y = x * 0.9;
}

void main() {
  float x = 1.0;
  float y = 100.0;  

  decimate(x, y);
  
  // x is still 1.0!
  // y is now 0.9!
}

When using out, the initial value of the argument is ignored, but it can be set by the function. The in qualifier used for the first argument is the default, so it is usually omitted.

Optional Arguments?

Before we answer that, I should note that names of variables must be unique. Using the same name more than once in the same scope will throw an error.

Functions have a slight exception to this rule: their arguments are part of the function signature, which means that you can declare multiple functions with the same name, as long as they have different arguments. Functions do not support optional arguments, but the same effect can be achieved using different signatures (albeit with some additional overhead).

float easeQuadIn(float t) {
  return t * t;
} 

float easeQuadIn(float t, float b, float c, float d) 
{  
  return b + easeQuadIn(t / d) * c;
}

In this example, I use two different signatures for an easing function. The first takes an argument t between 0.0 and 1.0, and returns an eased value. The second uses the four classic Penner easing arguments, and calls the first function internally. Because these functions have different arguments, their signatures are different, and they can coexist within the same scope.

The order in which these functions are declared is important. The second easing function uses the first, which would not be possible if they were declared the other way around.

Other than the number of arguments, the type of the arguments is also part of the signature, meaning you can declare multiple functions with the same name that operate on different types.

vec3 decimate(vec3 x){
  return x * 0.9;
}

Built-in functions

Aside from custom functions, GLSL also provides a number of functions that cover common mathematical operations. The types mentioned so far do not have any methods themselves. Instead you use the built in functions to operate on variables.

One of these functions is mix, which linearly interpolates between two values based on a float value between 0.0 and 1.0.

vec3 mixed = mix(v0, v1, 0.5);

Many of these built-in functions operate on several types. Check here for a full list of functions available in OpenGL. Note however that some of these may not work in WebGL. For a list of functions (and other things) that are definitely available, check the last two pages of this handy reference card.

Loops & Conditions

GLSL supports loops and conditional statements. The syntax for these is similar to JavaScript (but since GLSL is strongly typed, there is no need for the strict equality operator (===)). Using loops and conditionals is however generally discouraged, as it can create complications for how the shader is executed depending on your hardware.

Defines

Finally you may encounter defines. These are usually declared at the very start of a shader. They are used for values that are set once when the shader compiles, and do not change afterward. One of the defines Three.js uses is NUM_DIR_LIGHTS, which is set to the number of directional lights added to the scene. This is then used to correctly set the size of an array where these lights are stored.

#define NUM_DIR_LIGHTS 6;

uniform DirectionalLight directionalLights[ NUM_DIR_LIGHTS ];

In Three.js defines like this are dynamically determined and injected into the shader. If you want to add your own (using a THREE.ShaderMaterial), you can use the defines array in the constructor.

Shaders in Three.js

In WebGL, shaders are sent to the GPU as strings before being compiled. While this can be a little cumbersome to work with, it does offer a lot of flexibility. There are a number of ways you can deal with shaders, as long as they can be converted to strings in the end.

One way is to use HTML script tags, and send their textContent to be compiled. You can see an (old) example of this approach here.

Another is to use an array of strings, and concatenate them into a single string delimited by new-line characters. This is the basis for how shaders are used in Three.js. Three.js has multiple materials, all of which have their own corresponding shaders. However, some functionality is shared between different materials. To facilitate this, Three.js shaders are broken down into chunks, and combined in different ways for the final material shaders. My Three.js extension builds on this principle by injecting my own chunks into the mix.

If you can use ES6 for your project, you can use a multi-line template string instead of relying on arrays. This is a cleaner approach, and is much easier to maintain and edit.

Depending on your development environment, there are also ways to load standalone *.glsl files, and bundle their content with your JavaScript.

This concludes my brief overview of some of the features and constraints of GLSL. There are more things to know than covered here, but this overview should help you deal with some of the quirks of GLSL you may encounter when exploring the endless possibilities of shaders.

This is the second in a series of articles about advanced WebGL animation using Three.js and Three.bas, my extension for complex and highly performant animation systems.

In this post we will emulate part of the 3D graphics pipeline using plain old JavaScript and some of the utility of Three.js. There will be many omissions and simplifications, but this exercise should help you understand how some of the key elements fit together, without having to deal with everything at once.

If you want to skip ahead, the code we will be working toward can be found here. The whole thing is about 200 lines (excluding comments), so it should be pretty easy to step through even if you don’t read the rest of this post.

Our goal will be to draw a rectangle (or a Plane in 3D terms). We will defer the actual drawing to the canvas 2D API, but the steps to do so will be based on the WebGL graphics pipeline.

Like in Three.js, we will start by creating a scene, a camera, and a renderer. The scene will contain everything we want to render. The camera will determine our view of the scene. We will use THREE.PerspectiveCamera because it encapsulates all of the properties we need. The renderer will contain the bulk of the logic to make things appear on screen.

var scene = {
  children: []
};
var camera = new THREE.PerspectiveCamera();
var renderer = new Renderer();

Before we dive into the Renderer, let’s look at the model for our Plane, which will be represented by a Mesh with a Geometry and a Material.

Defining Geometry

3D models are represented by a list of “points” in 3D space called vertices. It is very important to realize that position is a property of a vertex, and not the vertex itself. Think of a vertex as a plain JavaScript object with a property called position.

Properties of vertices are called attributes. Alongside position, a typical vertex will have an attribute called a normal (a 3D point perpendicular to the surface of the model), used for lighting calculations, and a UV coordinate, used for texture mapping. Like JavaScript objects, vertices can have any number of attributes. This is limited only by the quality of your GPU, though you are much more likely to run into other issues before the number of attributes becomes a problem.

var vertex = {
  position: {x: 0, y: 0, z: 0},
  normal: {x: 0, y: 0, z: 1},
  uv: {u: 0, v: 0}
  // etc...
};

To keep things simple, our vertices will only have a position attribute. Our plane will have 4 vertices, centered around its own origin at {x: 0, y: 0, z: 0}. Let’s make it 200 units wide and 100 units high. This gives us the following:

var mesh = {
  geometry: {
    vertices: [
      // top left corner
      { position: {x: -100, y:  50, z: 0} },
      // top right corner      
      { position: {x:  100, y:  50, z: 0} },
      // bottom right corner
      { position: {x:  100, y: -50, z: 0} },
      // bottom left corner
      { position: {x: -100, y: -50, z: 0} }    
    ]
  }
};

Meshes are rendered in triangles called faces. Our plane will have two faces, and we will need to tell our renderer how to construct those faces from the vertices. This might seem somewhat arbitrary for a simple plane, but remember that all of this also applies to more complex geometries.

We will describe the faces using an array called indicies. Each number in this array represents the index of a vertex in the vertices array. Since we have 2 faces, the indices array will have 6 numbers (3 per face).

var mesh = {
  geometry: {
    vertices: [...],
    indices: [
      // first face
      0, 1, 3,
      // second face
      1, 2, 3
    ],
  }
};

Note that our faces are indexed clock-wise. While we will not take this into account for now, the winding of faces is very important in WebGL. Consistent winding enables WebGL to quickly determine whether a face is facing the camera or not. Faces that do not face the camera (the sides of an object you don’t see) are generally not rendered, boosting performance.

Defining Material

Geometry determines the shape of a mesh. The material determines its appearance. There are a number of different materials built in Three.js, all with their own set of properties. These properties are applied to all vertices. In our emulation, the material will only have one property: color.

In addition, the material will reference the shaders we will be using. In WebGL shaders are written in a language called GLSL, and sent to the GPU as strings. Our JavaScript emulation will use functions to approximate this process.

var mesh = {
  geometry: {...},
  material: {
    color: '#ff0000',
    vertexShader: basicVertexShader,
    fragmentShader: basicFragmentShader
  }
};

Defining Transformation

For the most part, geometries in WebGL should be considered static; once a model is loaded or generated, changing geometry data every frame will be very slow (and we want things to go fast). To facilitate animation, we will add 3 properties to our mesh that represent its transformation relative to the scene.

var mesh = {
  geometry: {...},
  material: {...},
  position: new THREE.Vector3(0, 0, 0),
  rotation: new THREE.Euler(0, 0, 0),
  scale: new THREE.Vector3(1, 1, 1)
};

We use Vector3 for position and scale, and an Euler for rotation. The Euler defines rotation around the 3 axes using radians (this is the same way you define rotation in CSS). These Three.js classes will make it easier to work with the math API later on.

The Renderer

As mentioned earlier, the renderer is responsible for drawing our mesh on screen. Since the mesh is defined in 3D coordinates, and the screen is flat, an important part of this responsibility is converting 3D coordinates to 2D. This process is called projection.

In order to correctly project a vertex on screen, we will need to take a number of things into account:

1. The position of the vertex relative to the model.
2. The transformation (rotation, scale, translation) of the model relative to the scene.
3. The transformation of the camera relative to the scene.
4. Projection properties of the camera, like field of view and aspect ratio.

This might seem like a lot, but luckily this process can be greatly streamlined using matrix math.

The section ahead deals with matrices and vectors (linear algebra). If you are not familiar with this topic, check out this article that covers the basics of matrices and vectors in Three.js.

Our render has one main method: render.

renderer.render(scene, camera);

This will render the scene as viewed through the camera. Let’s look at this method line by line.

Step 1: Clear the screen

this.ctx.fillStyle = this.clearColor;
this.ctx.fillRect(-1, -1, 2, 2);

This will clear the previous frame by filling the screen with a solid clear color. Since we are working in Normalized Device Coordinates (see previous post), we can clear the entire screen using fillRect(-1, -1, 2, 2).

Step 2: Update the camera

camera.updateProjectionMatrix();
camera.updateMatrixWorld();
camera.matrixWorldInverse.getInverse(camera.matrixWorld);

The projection properties of the camera, including field of view and aspect ratio, are represented internally in camera.projectionMatrix. This matrix needs to be updated whenever any of those values change between draw calls. The same applies to camera.matrixWorld, which represents the transformation of the camera relative to the scene. Because we are looking at the scene through the camera, we also need the inverse of camera.matrixWorld.

Step 3: Render each child

scene.children.forEach(function(child) {
  this.renderChild(child, camera);
}.bind(this));

Once the camera matrices are updated, we can move on to rendering the children of the scene. Since the steps for each child are the same, we can move this logic into a separate method renderer.renderChild(), passing the child in question and the camera along.

Step 3.1: Create a world matrix

var matrixWorld = new THREE.Matrix4();
var quaternion = new THREE.Quaternion()
  .setFromEuler(child.rotation);

matrixWorld.compose(child.position, quaternion, child.scale);

Inside renderer.renderChild() we first need to create a matrix representation of child’s transformation. This is similar to what we did for the camera. Matrix4.compose will help us do just that, but first we need to turn the Euler we used for rotation into a THREE.Quaternion to conform to the API for Matrix4.compose. After calling this method, we get a matrix that represents the combined transformation for our object.

Step 3.2: Create a model view matrix

var modelViewMatrix = new THREE.Matrix4().multiplyMatrices(
  camera.matrixWorldInverse,
  matrixWorld
);

Next we need a matrix that represents the transformation of the child relative to the position and rotation of the camera. In other words, this is how the camera sees the child. We call this a modelViewMatrix, composed by multiplying the inverse of camera.matrixWorld by the matrixWorld we just composed for the child.

var uniforms = {
  modelViewMatrix: modelViewMatrix,
  projectionMatrix: camera.projectionMatrix,
  color: child.material.color
};

The modelViewMatrix and projectionMatrix, alongside material.color will be used in our shaders as uniforms. Uniforms are values that are globally accessible in both the vertex shader and the fragment shader, which will come into play in the next step. We will also need the vertices and indices we set up earlier.

Step 3.3: Render the child face by face

var vertices = child.geometry.vertices;
var indices = child.geometry.indices;
var indexCount = indices.length;
var faceCount = indexCount / 3;
    
for (var i = 0; i < faceCount; i++) {

  // STEP 3.3.1: Retrieve vertices for this face

  var vertex0Index = indices[i * 3 + 0];
  var vertex0 = vertices[vertex0Index];
  
  var vertex1Index = indices[i * 3 + 1];
  var vertex1 = vertices[vertex1Index];
  
  var vertex2Index = indices[i * 3 + 2];
  var vertex2 = vertices[vertex2Index];

  // STEP 3.3.2: Project vertices using vertex shader
  
  var projectedVertex0 = this.applyVertexShader(
    child.material.vertexShader, 
    uniforms, 
    vertex0
  );

  var projectedVertex1 = this.applyVertexShader(
    child.material.vertexShader, 
    uniforms, 
    vertex1
  );

  var projectedVertex2 = this.applyVertexShader(
    child.material.vertexShader, 
    uniforms, 
    vertex2
  );

  // canvas 2D render logic
}

The code above renders a mesh face by face. First, the vertices corresponding to the current face are retrieved from geometry.vertices based on the indices array. Then material.vertexShader is executed once for each vertex in the face, returning a projected point in Normalized Device Coordinates.

Once all vertices in a face are projected, the area of the screen covered by this face is determined. Now the fragment shader takes over. The fragment shader is executed once for each pixel (fragment) in the area of the screen covered by the face. I couldn’t quite figure out how to effectively emulate this process, so we will just use the canvas 2d API to fill the shape using material.color.

this.ctx.fillStyle = this.applyFragmentShader(
  child.material.fragmentShader, 
  uniforms
);
this.ctx.fill();

What is a shader, anyway?

In the section ahead we will be looking at shaders and GLSL. If you want an introductory look into GLSL itself, check out this post.

WebGL shaders have two distinct but tightly coupled parts: a vertex shader and a fragment shader. These are essentially functions that can be injected into the graphics pipeline. Below is a very basic vertex shader written in GLSL.

attribute vec4 position;
uniform mat4 modelViewMatrix;
uniform mat4 projectionMatrix;
  
void main() {
  mat4 modelViewProjectionMatrix = 
    projectionMatrix * modelViewMatrix;
  
  gl_Position = position * modelViewProjectionMatrix;
}

This shader first calculates a modelViewProjectionMatrix using the supplied uniforms. This matrix represents the transformation for a vertex as seen by the camera, projected on a flat surface (the screen). This is the combined transformation for all of the steps we have taken so far. The shader then multiplies the vertex position attribute by this matrix, which gives us the position of this vertex on screen. gl_Position is a reserved keyword in GLSL, representing this final output value.

Each vertex has a different position, but the transformation represented by the matrix is the same (because it it based on uniforms). When this transformation is applied, each vertex is therefore transformed in the same way, so the shape (relative position of the vertices) of the mesh does not change. This is called an affine transformation.

Now let’s look at a JavaScript emulation of this shader. Below you can see each line and its JavaScript approximation.

function basicVertexShader() {
  // attribute vec4 position;
  var position = this.attributes.position;
  // uniform mat4 modelViewMatrix;
  var modelViewMatrix = this.uniforms.modelViewMatrix;
  // uniform mat4 projectionMatrix;
  var projectionMatrix = this.uniforms.projectionMatrix;
  
  // mat4 modelViewProjectionMatrix = 
  //   projectionMatrix * modelViewMatrix;
  var modelViewProjectionMatrix = 
    new THREE.Matrix4().multiplyMatrices(
      projectionMatrix,
      modelViewMatrix
  );
  
  // gl_Position = position * modelViewProjectionMatrix;
  return position.applyMatrix4(modelViewProjectionMatrix);
}

Since JavaScript does not support operators for custom objects, we need to use the Three.js math API to multiply our matrices and vectors.

Now let’s examine the applyVertexShader method we used earlier to project our vertices.

var projectedVertex0 = this.applyVertexShader(
  child.material.vertexShader, 
  uniforms, 
  vertex0
);

...

function applyVertexShader(shader, uniforms, vertex) {
  var context = {
    attributes: {
      position: vertex.position
    },
    uniforms: uniforms
  }

  return shader.apply(context);
}

To emulate how real shaders work, I created a context object which stores the values for attributes and uniforms expected by the shader. The shader is then executed against this context (using this to retrieve the supplied values).

While not exactly accurate, this is a fair representation of how shaders work on a high level. In JavaScript terms, shaders are stateless functions with a particular syntax for inputs and outputs. Inputs are represented by uniforms (which are the same for each execution of the shader within a single draw call) and attributes (which vary for each vertex). The output for the vertex shader is represented by gl_Position, which is essentially the return value of the shader.

This process is similar for the fragment shader, which is used in the next step of the pipeline. The fragment shader does not have access to vertex attributes, but it does have access to the uniforms. This is however where my emulation tapers off. You can see the complete code here, with some additional info in the comments.

If you are inclined to dig into the code and experiment some more, I created a little CodePen project that can be your playground. It’s mostly the same as discussed in this post, but the scene graph is a little more fleshed out, and there are utility classes to create different plane and polygon geometries.

There is much more to the graphics pipeline, but I hope this overview has given you new insight into how some important bits fit together. In the next post, we will dive deeper into vertex shaders, and examine how we can make use of the graphics pipeline to maximize performance.

This is the first in a series of articles about advanced WebGL animation using Three.js and Three.bas, my extension for complex and highly performant animation systems.

When I started working with 3D graphics development, one of the initial hurdles I had to overcome was understanding how the different coordinate systems through out the 3D graphics pipeline fit together.

This brief overview will disambiguate these spaces, helping you not to get lost along the way.

Pixel Coordinates

The first coordinate system is the one we are all most familiar with: pixel coordinates. It starts in the top left corner of the screen at {x: 0, y: 0}, and moves right and down to {x: viewportWidth, y: viewportHeight}.

While familiar, you will rarely be working with pixels directly. In fact, conversion to actual pixels on screen is the very last step in the 3D graphics pipeline. If you want a 3D object to show up at a specific location on screen, you will need to use trigonometry to calculate the corresponding position in 3D space first. I may come back to this in a later post, but for now let’s forget that pixels exist at all.

Normalized Device Coordinates

The second coordinate system is called Normalized Device Coordinates, or NDC for short. It starts in the center of the screen at {x: 0, y: 0}, with the x axis moving right and the y axis moving up. The range of values on both axes is -1 to 1. NDC is the same for all screens; whether you are using a tiny phone or an oversized flat screen, the values will not change.

If you have had to add pointer interaction to a WebGL project, you probably used a method like this:

window.addEventListener('mousemove', function(e) {
  var x = (e.clientX / window.innerWidth * 2) - 1;
  var y = (e.clientY / window.innerHeight * -2) + 1;
  
  // something awesome
});

This converts the position of the pointer on screen from pixel coordinates to NDC. You need to do this because NDC is an integral part of the math behind 3D graphics.

Normalized Device Coordinates are also the final output space for shaders (which we will get to in the next post). Therefor, when WebGL developers refer to screen space, this is the coordinate space we refer to, not pixels.

3D Coordinates

The third coordinate system is the actual 3D coordinates, often called world space or scene space.

By convention, red is used for the x axis, green is used for the y axis, and blue is used for the z axis. The color channels correspond to the axes; RGB is XYZ.

It starts in the center at {x: 0, y: 0, z: 0}. In Three.js, the x axis moves to the right (with negative values moving left), the y axis moves up (with negative values moving down), and the z axis moves towards the camera (with negative values moving away). This is called Right Handed Orientation. While common, this orientation is by no means standard. If you are importing models from 3D software, the orientation may be different. You then need to reorient the model by rotating it along one or two axes.

Other than orientation, you need to consider scale. If, for example, you create a 1 by 1 by 1 cube, what does that really mean?

The values along the axes in 3D space are essentially unitless; they can be whatever you want them to be. It is advisable to consider the scale of your world before you start building it. If you are working on human scale (cars, buildings), a 3D unit may represent a meter. If you are working with landscapes, it may represent a kilometer, and so on.

Being mindful of scale, and choosing and consistently applying the right dimensions will save you a lot of headaches down the road. If you don’t, you may find yourself having to work with values that are too big or too small to be practical. Even worse, you may have to go back and change a bunch of numbers throughout your code when things finally stop making sense.

UV Space

Finally I want to mention UV coordinates, used to map textures to 3D models. This coordinate system starts in the bottom left corner at {u: 0, v: 0}, and moves right and up to {u: 1.0, v: 1.0}.

If you are working with models generated in 3D software, or any of the Three.js primitives, UV coordinates will be generated for you. As such, you will not have to interact with them in code all that often. They may however show up if you are working with post-processing, where textures are used to transfer data between subsequent effects.

As mentioned before, Normalized Device Coordinates represent the output space for shaders. 3D coordinates (alongside UV coordinates and a whole bunch of other data) represent the input. In the next post, we will take a detailed look at how the conversion between the two happens.

This is a series of tutorials about WebGL, Three.js, and Three.bas, my extension for complex and highly performant 3D animation systems.

These tutorials are intended for people who know JavaScript and want to get into 3D graphics development, though some prior experience with Three.js will help.

We will start with a high-level overview of some key aspects of the 3D graphics pipeline. Then we will take a more detailed look at geometries, materials and particularly vertex shaders. Understanding how these work together on a hardware level will empower you to create beautiful animation systems without sacrificing performance.

Part 1: The Spaces of WebGL

Part 2: Emulating the 3D Graphics Pipeline

Part 3: Memory Management

Part 4: Form Follows Function

Addendum 1: Matrix Math and You

Addendum 2: A Brief Look at GLSL

The pieces themselves have two main components; a WebGL particle animation system, powered by THREE.BAS, my THREE.js extension for shader based animation, and a masking/compositing approach. The repo for THREE.BAS has some documentation, examples and a brief explanation of the thinking behind it. I may further elaborate on it at a future post, but today I will focus on masking and compositing.

Alpha Masking

An alpha mask is a grayscale image that is used to mask out content with degrees of transparency. Below, you can see the original image and the alpha mask that I used for the WebGL content.

When the mask is applied, pixels under the white area are rendered fully opaque, and pixels under the black area are rendered fully transparent. The grays in between create an alpha gradient for smoother blending.

You can see the final composition below. I simply place a WebGL canvas over the cinemagraph, and apply the alpha mask to it.

There are a number of ways we can use alpha masks on the web, both inside WebGL/THREE.js and through CSS. Because support for CSS image masks is still iffy, I decided to add the alpha mask as a post processing step in THREE.js.

Digging into how shaders/post processing works goes well beyond the scope of this post, but this tutorial gives a good introduction to the concepts behind it.

This is code for the alpha mask post processing step:

var maskPass = new THREE.ShaderPass({
 uniforms: {
 // the underlying image
 “tDiffuse”: { value: null },
 // the alpha mask
 “tMask”: { value: null }
 },
 vertexShader: [
 “varying vec2 vUv;”,
 
 “void main() {“,
 “vUv = uv;”,
 “gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);”,
 “}”
 ].join(“\n”),
 fragmentShader: [
 “uniform sampler2D tDiffuse;”,
 “uniform sampler2D tMask;”,
 “varying vec2 vUv;”,

“void main() {“,
 // get the current pixel color
 “vec4 texel = texture2D(tDiffuse, vUv);”,
 // get alpha based on the color of the mask image
 “float alpha = texture2D(tMask, vUv).r;”,
 // apply the alpha to the current pixel
 “gl_FragColor = texel * alpha;”,
 “}”
 ].join(“\n”)
});

It’s based on the THREE.CopyShader, which simply passes through the contents of the screen to a subsequent post processing step. Instead of passing the pixel values directly, the mask pass reads an additional texture (the alpha mask) and applies an alpha based on the red channel of the mask at the same UV coordinates.

// get alpha based on the color of the mask image
“float alpha = texture2D(tMask, vUv).r;”,

This works because the mask image is grayscale, so the RGB values are all equal. In shaders, color channels are interpreted as numbers between 0.0 and 1.0. Multiplying a pixel color by 0.0 will make it fully transparent, while multiplying it by 1.0 will not change it at all.

Making a scene

The alpha mask approach creates some interesting possibilities, but it has limitations if you want to treat the cinemagraph as an actual three-dimensional space, like below.

It should be possible to have the 3D objects pass both behind and in front of the statue using masks, but this makes things needlessly complicated, and may involve rendering the scene several times. Instead, I simply added a 2D cut-out of the statue in the image to the THREE.js scene, and made the particles swirl around it.

Below is the same composition without the underlying image, and the mouse camera controls enabled.

The only tricky part here is determining where to place the 2D cut-out image in the 3D scene, so that it has the same dimensions as the image in the DOM. Basically, we need to calculate the size of a plane at a given distance from the camera, such that the plane is exactly the same size as our viewport (which is the same size as the image behind it).

Fortunately, this is pretty easy as long as the camera is looking straight at the scene:

var cameraZ = camera.position.z;
var planeZ = 5;
var distance = cameraZ — planeZ;
var aspect = viewWidth / viewHeight;
var vFov = camera.fov * Math.PI / 180;
var planeHeightAtDistance = 2 * Math.tan(vFov / 2) * distance;
var planeWidthAtDistance = planeHeightAtDistance * aspect;

This formula uses the camera field of view (fov) and the aspect ratio of the viewport (width / height) to calculate the desired size for the plane on which the cut-out image is rendered as a texture.

All that’s left to do is make the 3D particles rotate around the same Z coordinate as the plane, and you’ve got yourself a scene.

Depth masking

Further iterating on this, I wanted to apply the same concept, but with an added layer of depth. This resulted in the composition below.

Here the 3D particle swarm starts behind the back building, swirls behind the front building, then moves in front of the back building, before going off screen in front of the front building. This effect is achieved by creating a little mock scene over the original image, matching (somewhat) in perspective and size. Once that is in place, coordinating the motion and layering becomes much easier.

Below is a debug version of the same composition, with the mouse camera controls enabled again.

The two white planes that represent the buildings in the 3D scene are not rendered in the final composition. I could have textured them using the same approach as the Bruce Lee statue, but this would have been harder because of the camera position.

Since the actual buildings are already in the image, the only thing we care about are their dimensions and position in the 3D scene. Because of this, we can discard the color buffer after the buildings are rendered, as long as we keep the depth buffer intact.

// clear color and depth
renderer.clear(true, true);

// render the two buildings
renderer.render(buildingScene, camera);

// clear color (but not depth)
renderer.clear(true, false);

// render the particles
renderer.render(particleScene, camera);

With this approach the planes that represent the buildings essentially become a depth mask for the particles; invisible walls for them to move behind.

What’s next

I’m pretty excited to have stumbled upon this format. Codevember is far from over, and I intend to make at least a couple more contributions. One area I want to explore is timing the WebGL content to the cinemagraph. Unfortunately, this is pretty much impossible with gifs, but I can use video or a png sequence for both the cinemagraph and the mask to achieve some cool effects.

WebGL Masking & Composition was originally published in Your Majesty on Medium, where people are continuing the conversation by highlighting and responding to this story.