Custom Materials with Google Sceneform and ARCore

10 min readJun 26, 2018

Announced last month at Google I/O, Sceneform is an easy-to-use 3D rendering solution for ARCore. The recent 1.3 release added a bunch of new features including the ability to write custom material definitions composed of OpenGL ES fragment and vertex shaders.

In this post, we’ll walk through writing a custom material definition and use it to render an object with Sceneform. We’ll keep it minimal, only scratching the surface of what’s possible, but we’ll cover the basics — authoring a material with a fragment shader, setting material parameters, texture sampling, and physically-based rendering. Be sure to check out the Custom Material Reference for more information.

Who should read?

If you have your own GLSL shaders that you want to use with Sceneform or you’re curious about writing your own, this is for you. Note that Sceneform refers to these as material definitions, but as you’ll see, they’re basically a GLSL shader with some additional metadata.

It helps to have some experience with Sceneform and Android Studio. If you’re completely new to Sceneform, you might want to check out these resources to get you started:

Basic shading concepts and familiarity with a shading language like GLSL is also helpful. This quickstart is for advanced users who are interested in writing their own materials.

Prerequisites

You’ll need Android Studio to follow along. I’m using Android Studio 3.1.2 for the purposes of this tutorial.

Be sure to also install the Google Sceneform Tools (Beta) plugin for Android Studio, which allows you to import and preview models directly in the editor. You can follow the steps here to get it set up. If you’ve already installed it, be sure you have the latest version (1.3).

For the purposes of this tutorial, I’ll be walking through updating the Hello Sceneform demo app from the Sceneform SDK for Android to use a custom material. You can follow along with the Hello Sceneform app, or use your own. If you've cloned the Sceneform SDK previously, make sure to do a git pull to get the latest updates. You can also download a zipped version.

If you’re using your own project, check that your build.gradle file has the latest version of the Sceneform Gradle plugin (1.3.0):

dependencies {
    classpath 'com.android.tools.build:gradle:3.1.2'    // Version 1.3.0 required for custom materials
    classpath 'com.google.ar.sceneform:plugin:1.3.0'
}

Exploring the SceneformSDK sample

If you’re following along with the Hello Sceneform project, go ahead and open it up in Android Studio.

Hello Sceneform is a minimal ARCore app that shows the basics of the Sceneform SDK. The app is pretty simple: you tap to place green Android “Andy” models on an AR surface. The app is bundled with an Andy model and texture which is used by Sceneform to render the Andys. Sceneform has a few materials it uses as default, but the goal for this tutorial is to write our own custom one from scratch.

Take a look in the app/sampledata/models folder, and you'll see the Andy model (andy.obj) and texture (andy.png):

You’ll also see an andy.sfa file. This is a Sceneform Asset Definition. This file is a human-readable description of the models, materials, and textures used to render the AndyRenderable. For more information on Sceneform Asset Definition files, take a look at the documentation.

Taking a look at andy.sfa

First, a quick note on Sceneform file types: the sfa extension stands for Sceneform Asset. If you look inside the app/res/raw folder, you'll see a corresponding andy.sfb file. sfb stands for Sceneform Binary. The sfa file is human-readable and editable while the sfb file is a binary format, and is what is used directly in your app. The sceneform.asset declaration in your Gradle file adds a task that generates the sfb file from your sfa file when you do a build. When you open up sfb in Android Studio with the Sceneform Tools plugin enabled, you'll actually get a view into the sfa file. When I refer to editing the andy.sfa file, just know that you can make the edits through either Android Studio view— either the sfa or corresponding sfb file. That being said, in order to see the model in the Viewer, you'll need to have the sfb open.

Go ahead and open up andy.sfb inside Android Studio. If the model viewer isn't automatically opened, open it up by navigating to View -> Tool Windows -> Viewer.

This viewer will update as we edit the sfa file and our custom material, so keep it open! You can click and drag in the viewer to orbit around the model.

The andy.sfa file in the project was generated with one of the default materials. Since we're aiming to use our own material, we're going to regenerate the sfa file.

Creating a custom material

First we’re going to write a boilerplate material definition. Custom material definitions in Sceneform have the .mat extension. The first step is to create a new mat file next to andy.obj, in the app/sampledata/models directory. In the Android Studio project view, right-click on the app/sampledata/models folder and select New -> File. Name it custom_material.mat. Be sure to use mat as the extension. Android Studio might ask you to associate the file type— just select Text File. Copy and paste the following material template into the file:

We’ll get into what each section means in a minute. Here’s what your directory structure should look like now:

This is a minimal material definition that just sets Andy to an unlit, solid color — Not too interesting, but it’s a start! Let’s use it with the Andy model.

Using the custom material for the Andy renderable

Since the project already has Andy set up to use a default material, we’re going to delete and recreate andy.sfa, but this time hook it up to our new custom material.

First, delete the andy.sfa file from the project by right clicking on it and selecting Delete.... Android Studio might give you a warning that the file is being referenced elsewhere. That's okay, just hit Do Refactor.

Now head to the build.gradle file for the app module. You'll see the sceneform.asset towards the bottom:

sceneform.asset('sampledata/models/andy.obj',
        'default',
        'sampledata/models/andy.sfa',
        'src/main/res/raw/andy')

The second argument, 'default' specifies the material to use for the asset. default, of course, means to use a default material appropriate for the model type (obj, fbx, etc). Instead, let's point it to our custom material, like so:

sceneform.asset('sampledata/models/andy.obj',
        'sampledata/models/custom_material.mat',
        'sampledata/models/andy.sfa',
        'src/main/res/raw/andy')

Save and rebuild the project if necessary, and you should see a new andy.sfa file appear in app/sampledata/models/, right where the previous one was. You may also need to close and reopen andy.sfb to see the changes. The preview should update with a colorless Andy:

The lack of color is because we specified a material parameter, and by default it’s 0, 0, 0. Open up andy.sfa and find the andyColor field under parameters. Change it to whatever you like— I choose yellow:

parameters: [
   {
      andyColor: [
         1,
         1,
         0,
      ],
   },
],

Do a save, and watch Andy’s color update.

Congrats! You just wrote your first custom material. It’s not much yet, but you can probably see the awesome potential we’re wielding.

If you’re an advanced user, the Custom Material Reference might be enough from this point to get you started. Read on and I’ll dive deeper into the material definition format and we’ll give Andy a texture.

The material definition file

Let’s look a little closer at what we copied into our custom material. Sceneform material definitions are JSONish and have two required blocks:

material {
    ...
}fragment {
    ...
}

The material section sets some property pairs for the material, like the name, shadingModel, and material parameters.

The name property gives the material a name for debugging purposes. We don't need to worry too much about it.

The parameters property specifies a single material parameter: a float3 named andyColor. I'll talk more about material parameters in a bit.

The requires property denotes which vertex attributes are required by the material. For now, we only need position, which is the vertex position data. This is included by default, but I added it for completeness.

The shadingModel sets the overall algorithm used to calculate lighting. There are a few different shading models we can use, which are outlined in the Custom Material Reference.

The fragment section is where we write the fragment shading code to shade our material. We are required to define a function material that takes a single MaterialInputs argument named material.

void material(inout MaterialInputs material) {
    prepareMaterial(material);
}

The only requirement of the function is that it call prepareMaterial(material) before exiting. From the documentation:

Note that you must call prepareMaterial(material) before exiting the material() function. This prepareMaterial function sets up normal related internal state. Some of the APIs described in the Fragment APIs section - like shading_normal for instance - can only be accessed after invoking prepareMaterial().

This is where we put our GLSL fragment code. Here we’re just setting the baseColor property, which represents the albedo of the material, to the andyColor material parameter. Sceneform has a full physically-based rendering system which we access through setting fields on the material variable. If you change the shadingModel to "lit", you can try setting additional properties, like metallic and roughness. You can read more about Sceneform's PBR and material properties here.

Material parameters

If we wanted to, we could have hard-coded the color right inside the fragment shader, using a float3(1.0, 1.0, 0.0) constant. We want to be able to programmatically set values in our shader though, so we're using Material parameter. You may have already used these if you have experience with Sceneform.

The parameters field in custom_material.mat sets up material parameters for our material:

parameters: [
    {
        type: "float3",
        name: "andyColor"
    }
],

We can then use the material parameter inside the fragment code via the materialParams variable:

material.baseColor.rgb = materialParams.andyColor;

Since we gave the parameter a name of andyColor, we can access it via materialParams.andyColor.

The nice thing about material parameters is that we can also set them programmatically in our app without needing to recompile the material. If you want, edit HelloSceneformActivity.java and programmatically set the andyColor parameter:

This way, you can update the color inside your app dynamically. See the parameters documentation for all the different types of material parameters. We’ll do one more example with a special type of parameter, a sampler2d, in order to use an image as a texture.

Adding a texture

Instead of using a solid color, let’s texture Andy with the andy.png image. First, change the type of the andyColor parameter in our custom material from float3 to sampler2d.

parameters: [
    {
        type: "sampler2d",
        name: "andyColor"
    }
],

The sampler2d provides us with a GLSL sampler inside our material function. The only difference is that we access it using the materialParams_andyColor notation (notice the underscore instead of a dot).

Update the material function:

void material(inout MaterialInputs material) {
    prepareMaterial(material);
    float3 texSample = texture(materialParams_andyColor, getUV0()).rgb;
    material.baseColor.rgb = texSample;
}

Here we’re using a new function called getUV0(). This allows us to access UV coordinates inside the fragment code. Then we use the GLSL texture function to sample the texture (we'll provide the sampler with a data source in a moment).

Before we can use the getUV0 function, we have to specify that our material requires UV coordinates. This is done by adding another value to the requires section:

requires: [
    "position",
    "uv0"
],

The only thing that’s left is to hook the parameter up in the sfa file. Before, we were specifying a float3 to use as andyColor, but now we want to feed it an image sampler. If you're working off the Hello Sceneform demo, you'll notice a samplers section in the sfa file:

samplers: [
   {
      file: "sampledata/models/andy.png",
      name: "andy",
      pipeline_name: "andy.png",
   },
],

This is added because the model was imported to the project from an obj file, which specified a png texture. Note how the name field is set to andy. Replace the andyColor parameter in the sfa file with a reference to the sampler:

materials: [
   {
      name: "unlit_material",
      parameters: [
         {
             andyColor: "andy"
         }
      ],
      source: "sampledata/models/custom_material.mat",
   },
],

The samplers section is where you could set up min and mag filters and wrapping types for the sampler. Again, see the documentation on the sfa file for more information.

Save, and Andy should update with a texture:

Physically-based rendering

For reference, here’s where we’re at with the material definition:

and the andy.sfa file:

I’ll leave you with a taste of Sceneform’s physically-based rendering capabilities.

Let’s mess around with the baseColor, metallic, and roughness properties.

First, make sure shadingModel is set to lit. I wanted an additional texture to make things a bit more interesting, so I grabbed the painted-metal-02 texture pack from www.cgbookcase.com. I decided to use the Painted_metal_02_Metallic.png file. You might want to scale it down a tad, as the full resolution version is 4K.

I added it to the Android Studio project, and added another samplers entry in andy.sfb

{
   file: "sampledata/models/Painted_metal_02_Metallic.png",
   name: "metal"
},

Add a new sampler2d parameter to custom_material.mat called metal, hook it up to the newly added metal texture, and try out this material definition:

A weathered Andy- turns out he’s made of gold

I’ll leave it as an exercise for the reader to work through the details. I highly recommend https://thebookofshaders.com/ as a learning resource when writing your own materials.

This is only just scratching the surface of what’s possible with Sceneform. Sceneform has a lot more to offer, including a physically-based renderer with lit, cloth, and subsurface shading models. Be sure to check out the full Custom Material Reference.

Thanks for reading! Hope it was helpful — feel free to comment or tweet at me with questions.