Dynamic textures in Sceneform

Recently Android Graphics team presented a cross platform, real-time physically based rendering engine for Android, iOS, Linux, macOS, Windows, and WebGL— Filament.

Then, on top of it, they’ve built Sceneform — library that in pair with ARCore, helps to render realistic 3D scenes in AR and non-AR apps, without having to learn OpenGL.

While Sceneform is a pretty fresh library, it’s already has a lot of great functionality and supports few common 3D model formats, including:

• *.obj
• *.fbx
• *.gltf

Though, things requires some dancing around. To include your models to the projects, you still need to convert them to *.sfb binary format, using Android Studio plugin.

Along with *.sfb file, Android Studio generates a *.sfa text file, that describes how to build binaries in human readable form. This file can be changed manually, and changes will be applied to binaries with next Gradle sync.

Later on, you can load those compiled binary files, both statically from res or asset folder and dynamically from file or network storage using URI’s:

// Get Rendarable from assets, using URI
val renderable = ModelRenderable.builder()
    .setSource(this, Uri.parse("$name.sfb"))
    .build()

In one of latest releases, Sceneform Assets library was introduced, that also allows you to load glTF or glb models with out additional conversation.

Material Definitions

Material definitions are some sort of Shaders in Sceneform. By default there are three of them, one for each supported model type:

• OBJ assets: obj_material.sfm
• FBX assets: fbx_material.sfm
• glTF assets: gltf_material.sfm

They describe what parameters are allowed during asset generation, and lately can be changed on runtime. Depending on what asset type you are importing, corresponding material definition will be used.

That’s for instance, default Material Definition for OBJ assets, have possibility to change :

• baseColor — defuse texture.
• baseColorTint — applies a tint to the computed baseColor value, specified as [r, b, g, a] vector 4 value.
• metallic & roughness — Controls the metallicity & roughness of the material. Float value from 0 to 1.
• opacity — pass null to make model fully opaque, or float value from 0 to 1 to make it transparent.

If you want to apply metallic , roughness and normal texture maps to the model, you shall consider of using FBX or glTF assets instead, they support that out of box.

• Material Definition for FBX assets has normalMap , metallicMap and roughnessMap parameters.
• Material Definition for glTF assets has metallicRoughness and normal parameters.

Dynamic textures

To get texture dynamically, use Texture.builder() class, where you need to pass source of your asset and usage type with one of Texture.Usage constants (COLOR — for color(defuse) textures, NORMAL — for texture of normal and DATA — for everything else, including metallic and roughness):

val texture: CompletableFuture<Texture> = Texture.builder()
    .setSource(context, Uri.parse(uri))
    .setUsage(Texture.Usage.DATA)
    .setSampler(
        Texture.Sampler.builder()
            .setMagFilter(Sampler.MagFilter.LINEAR)
            .setMinFilter(Sampler.MinFilter.LINEAR_MIPMAP_LINEAR)
            .build()
    ).build()

Then, you can get Material from your Rendarable, and change its texture with renderable.material.setTexture(“parameterName”, texture) method.

parameterName — property, represents one of properties defined in asset Material Definition.

Different Model flavors example.

Im my case, we had a pure OBJ asset mesh, with out any texture reference in it. And near them, we got different sets of texture assets, so for instance we can have two flavors of the same object but with different colors and material structure.

First, we tried to convert our OBJ to both FBX and gLTF models, and was expecting that everything will work good, as those types do have texture maps support.

But, there’s always a but. If you have a model, with out texture references inside of it, you will be not able to change them with material.setTexture method.

After searching the GitHub, we’ve found few related issues:

• How to change textures in runtime? #41
• How can I change metallic or roughness factor in runtime? #112
• Change scale and proportions of renderable at runtime #197
• fade mode layering wrong #390
• How to dynamically change normal, metallic and roughness with Textures? #412

Dummy Object

We’ve discovered, that to change the texture, we need to use fake object and its Material:

• Create a fake model, with faked texture references (point to some default assets);
• Import that fake model, and convert it to sfb ;
• Include it to the project, the best way will be to bundle it to the APK, one of res or assets folder;
• Get Material copyRenderable.material.copy() from that model;
• Apply your changes to Material, with setTexture method and set it to your real object.

In theory you can create fakeObject.fbx triangle or cube with references to some dummy default texture, using some 3D redactor like Blender. Then get it in application, set textures and apply to real FBX imported object.

But that’s theory. On practice, for instance, Blender cannot export FBX with embedded textures. We’ve tried to manually update the SFA file with some textures, but in result our textures where not applied to compiled SFB.

Custom Material

And thats were Custom Material definitions come. Sceneform allows you to define and use your own material definitions instead of default one.

Making and using it in pair with dummy object, was the only one working solution that we’ve currently found.

Here’s the *.mat file:

Custom Material Definition for dynamic textures.

Now you can generate fakeObject.sfb using this custom material. In gradle.build file:

sceneform.asset('sampledata/fakeObject.obj',
        'sampledata/pbr_material.mat',
        'sampledata/fakeObject.sfa',
        'src/main/res/raw/fakeObject')

*pbr_material.mat —link to custom material.

And realObject.sfb using default material:

sceneform.asset('sampledata/realObject.obj',
        'default',
        'sampledata/realObject.sfa',
        'src/main/assets/realObject')

The last thing is to apply fake object’s material copy to real object:

Applying textures to Renderable dynamically.

I’m sure Android Graphics team will polish some rough corners of the SDK in future, till then, hope this workaround will help you to dynamically load textures with Sceneform.

Sample project

You can find sample project source code here.

Other

• MaterialBuilder public constructor #393 — subscribe to track progress for adding material creation feature;
• Name of material #315 — subscribe to track get name of the sub materials features.