MatCap — Render & Art pipeline optimization for mobile devices

Charles Barros

Published in

PlayKids Tech Blog

8 min readNov 29, 2019

TL;DR

Download the MatCap shader and get free MatCaps from Pixologic [1] to start using it in your project! 😉

Intro & Motivation

In a world where PBR Shading and Post Effects are a real part of AAA Desktop games, I invite you to take a deep breath, jump out of the hype train and embrace the reality that most mobile devices can’t deliver the performance to use these techniques.

One of the most important aspects of a mobile game is how broad its target audience is, so it probably won’t be a good idea to have your game running only in high-end devices. That should be no surprise for anyone, but there are several non-obvious performance aspects that novice developers usually underestimate.

Let’s say that you just released a new mobile game with cutting edge graphics, targeting high-end devices. Although in your tests everything runs smoothly, you may notice a lot of bad reviews in the app stores complaining about low performance. You will probably check your crash report tool, trying to figure out what’s wrong with it and you won’t be able to find out any significant clue.

During game development, we run the game every day, but in a different way that our players will do. You will probably jump directly to the feature that you are working on, test a few use cases and go right back to coding. It’s time to think more about how your users consume your app, rather than only thinking about GPU specs and Shaders instruction count.

Your users will actually play your game, and luckily, some of them will love it and play it for hours! What about you? Have you ever played your game on a real device? If this sounds surprising to you, know there’s a lot of developers that don’t test their games on real devices. So, take some time and play your game on your own device for at least one hour. I guarantee that the hours spent will be invaluable to set your mindset throughout our journey.

The first thing you’ll notice is that the game appears to run smoothly, but after some time performance may decrease. Probably you want to say hello to our little friend, thermal throttling [2]! The device that you are holding is a fantastic product of engineering, with so many things packed in such a small device. To achieve such technical prowess some compromises had to be taken. There is no room for active cooling mechanisms, so after a long session of flatout load your high-end device may start to get hot, and to avoid hardware failures, it will slow down the chipset clocks.

Another issue you might encounter is how bad it is to hold a burning hot device using your bare hands. Your hands will start to take turns and before you know it, you won’t even be able to hold your device, thus you will be forced to end your session, not because you want to, but because you are having a bad experience.

Last, but not least, after closing your game, look at your device’s battery level. Notice how this game session hurt your battery and imagine that you must make it last for the rest of the day. That means no more game sessions or any sort of phone activity until you reach a power cord.

Do you really want the most engaged users of your game to leave their sessions prematurely and remember it as a bad experience? They spent a lot of money buying a top-notch device, probably spent some money in your game and they deserve the best possible experience.

Now, I could be writing articles about techniques to avoid thermal throttling, ways of using different instruments to check battery consumption, but I choose to take a different route.

Many of our games are developed using a simple and very powerful shader technique called MatCap. This technique allows you to deliver high-quality graphics while avoiding exhausting all of the device resources during your game session. In this article, I’ll explain some of the advantages and limitations of this technique and why you should consider it.

MatCap in a nutshell

MatCap stands for “material capture”. It was popularized around 2007 in Zbrush software[3], but it seems to have earlier unknown origins.

Different from other shaders that try to calculate the way that the light sources interact with the surfaces, MatCap is a clever and non-realistic way to simulate these interactions by encoding every aspect of a lighting setup and material properties into a single texture.

Let’s say that we are trying to simulate a very common photography lighting setup called three-point lighting. So first we have to bake the lighting and shading (mainly diffuse & specular) to our main actor, this glorious sphere. The result would be something like this:

Figure 1 — Three-point lighting setup encoding

Now it’s time to encode other aspects of the material itself, like how it reflects the environment (smoothness & fresnel):

All we have to do is to crop a bounding box around the sphere and we will have an element that indirectly recorded all these settings.

Now we take this image to a shader that will use all the data encoded into the MatCap texture to simulate the same looks into an arbitrary 3D model like this house in our game PKXD:

Figure 4 — MatCap texture applied to custom 3d models. Cell-Shading, metallic reflections, everything rendered with the same shader.

There are several ways to generate the MatCap image:

Offline rendering (3D Studio, Maya) [4]
Real-time rendering using a game engine
Real photography setup and a chrome sphere
Artistically painted by hand.

This technique enables you to navigate from a very realistic look and feel, to a cartoon/cell shading only by changing one texture. So let’s make it clear: MatCap is a technique that simulates shading, but does not use real-time lighting sources. It’s weird to think about it, but technically it’s an unlit shader.

You can, of course, to some extent, generate these MatCap images in run time to have some control over the lighting setup, but you will not get accurate light simulations.

How to shade an object using a MatCap texture?

Simply put, MatCap is a spherical environment mapping. Sphere mapping [5] is one of the simplest reflection mapping techniques and was very popular back in the early 2000s. The reflection is not realistic since the spherical mapping causes distortions and the reflection image is not updated in real-time, but it’s a good tradeoff between quality and performance.

The main concept regarding sphere mapping is to use the object’s local normal vectors as UV mapping coordinates of the MatCap texture (the Math involved will be explained later in this article). The desired effect is that when the camera moves around the object, the shading (highlights, shady areas, and reflections) move around your object.

Wouldn’t it be nice if we could use this technique to simulate every aspect of a given material, and not only their reflection? That’s exactly what MatCap does!

Figure 5 — Different looks by just changing the MatCap texture

Spherical Environment Mapping in depth

For the sake of simplicity, I will use the following Unity Shader to explain the algorithm behind sphere mapping and MatCap.

Matcap.shader

Basically, in the vertex function, we should transform the vertex normal to mesh space by multiplying it by the unity_WorldToObject vector and the result by the view matrix UNITY_MATRIX_V that transforms from world space to local view space. Confused about local view space? Fear not, let’s see what’s Unity documentation has to say about it:

“This is similar to if you had a gameObject as a child of a camera gameObject, but without any scale applied
This means the positions are all in world space distances, but with the camera at 0,0,0 and rotated to match the camera’s orientation” [6]

Ok, but there is something weird inside of the vertex function. Why we are multiplying the result by 0.5 and summing 0.5 to it?

Let’s think about the normal vector. It’s a 3D vector where its values range from -1 to 1. We are planning to use it as UV coordinates to map our MatCap texture. The UV range goes from 0 to 1 (keep in mind that for this effect UV tilling does not make sense), so the vertex function will simply convert from range -1..1 to 0..1:

The last part is just a little tricky, which involves blending the diffuse texture to the MatCap. It’s like the overlay blend function in Photoshop. Since we are using the MatCap texture to create shading (shadows and highlights), the blend function must be able to darken and lighten the diffuse texture. We are just doing the opposite of the operation that I previously explained. This will convert from range 0..1 to -1..1:

For better results in low-poly models, you may use the normals from a normal map instead of the mesh normal vectors. This will be significantly heavier since all the space transformations will occur in the fragment method, but it is still lighter than regular pixel-lit shaders.

Conclusion

This technique empowers artists to create many different materials without the need to create custom shaders. It saves time during game development and it makes it possible for the artist to achieve the desired result by just shading a sphere with the aspect that he wants in the final object.

The overall rendering process is very optimized because we are just sampling a MatCap texture and using the result of a sphere mapping process as UV. We can encode dozens of effects in the MatCap texture, like hard shadows, multiple light sources, reflections, and the performance will be constant.

It is important to state that MatCap makes it possible to achieve a good look and feel for several types of games and it makes it run smoothly in a broad range of devices, but it is not suitable for every genre of games. It has many limitations since lighting will probably be inconsistent along the objects and it’s hard to update the lighting setup in real-time.

Many other cool tricks can be accomplished with MatCap, like simulating different materials using just one draw call or creating awesome alpha-blended effects. Want a sequel of this article? Let us know in the comments.