How to improve performance in Augmented Reality applications — Part 1

Published in

XRPractices

6 min readAug 25, 2020

--

Almost all AR developers have faced the issue of low frame rate, devices losing all the battery within minutes into the application. I have been in situations where I have been asked to make an app more real and smoother. Sometimes, I have encountered augmented reality applications that make devices heat up within a few minutes.

We will see below what are the reasons behind such issues and how we can fix them.

It is a tough choice to make between beautiful and smooth, as to make it beautiful there will be more calculations and hence it will impact the frame rate thus decreasing the overall experience.

I hope this article will help to achieve AR-app performance goals, if you just want to read the tricks then navigate directly to “Key Take-Away” section below

Disclaimer — I will be discussing the tricks and tips that can be used with an application built in Unity3D and if you have used Unreal/Android Studio/XCode for the development even then you can read about the reasons behind the underperformance of your app and then apply the strategies in your development environment as the issues are same across all AR-apps build in any development tool.

We will see the strategies that we can apply in the overall development cycle.

Conceptualization/Planning

There are questions that we should ask from the client, like whether they will provide us with the 3D models or not. And, if yes, then most probably it will be a CAD file. Go through the entire interaction in your head or draw them on the whiteboard. This will help you to visualize the visible portion of the 3D model and ask your 3D artist who is either creating the 3D art or optimizing the CAD file to delete everything that is hidden, YES, Everything!

This will reduce the calculations and hence solve the issue of overheating of the device.

Now, this is what most of the developers must have already done. But, there is one more additional step that will give you that last push. Try to use opaque textures… No, I changed my mind! Use opaque textures and not transparent materials as that will reduce the additional calculations, i.e. overdraw — we want to keep the overdraw minimum in every way possible as transparent materials use more GPU resources for rendering

Another important guideline for the artist in the team is to create atlases for the textures and map UV accordingly this will help enable batching which we will see below in more detail.

Key Take-Aways for 3D Artist/Generalist/Technical-Artist —

1. Remove all the hidden mesh — Delete ’em all.

2. Use Opaque textures — Yes you can use transparent textures but only if you have to.

3. Keep UV mapping seams and hard edges (doubled-up vertices) as low as possible.(Developers, not for you 😅 — for curious minds)

4. Create a single texture map/atlas that can be used for all the materials in the 3D model (For batching — discussed in detail below)

Development

During development as a developer, we have to make sure that a lot of things help to achieve the targeted performance for the targeted platform, this is where Unity3D profile r comes in handy. Profiler lets us see the actual calculations when the app is running on the device and we can see what is bottle-necked, it can be either CPU, GPU, or memory.

For each, we have different techniques and we must allocate a budget for each before we start optimizing for e.g. optimization of memory may bring more load on CPU or GPU and vice versa e.g. mip-maps improve the frame rate but at the cost of eating up more memory

GPU optimization techniques —

Always check the triangles that are rendered in the scene, as above, I highlighted that deleting all the invisible meshes is the reason for it. More triangles mean more calculations by GPU.

Since the AR applications have limited focus areas,i.e., in an AR app we are focusing on a particular/single 3D object so always adjust the far clipping planes of the camera to the minimum as per your needs of the application. This will help in saving a few calculations as well. ( we are discussing the view frustum here)

I am skipping occlusion culling intentionally as this article is for the augmented reality apps and occlusion culling primarily comes into play when we are developing triple-A games or virtual reality applications. I am also skipping the lights optimization as generally we have one light per scene in the case of AR. So it won’t make much difference.

Another way to give some breathing space to your GPU is by compressing the textures as per your platform. I won’t go into details for it as you can refer to this article by Unity3D

Use mips-maps of textures as this will reduce the rendering time and will boost your frame rate but at the cost of eating up your memory so if you have other plans for the memory then I suggest you make a budget for mip-maps.

CPU optimization techniques —

Here comes a fancy word that we all love to use to justify why our application sucks so bad.

Bad here means battery draining too quickly, device heating up, and the obvious one “The low Frame-Rate!”

So what is a Drawcall — you can see above in the image above, I have highlighted the draw calls. Actually, they are the commands that tell the GPU to render a certain set of vertices as a triangle in a certain state(shaders, blend state, blah blah…) this causes performance overhead on CPU. For a deep understanding of drawcall please see this video, it is quite old but it has all the explanations about drawcall.

For good performance, we have to keep the draw call as low as possible as a group of triangles with the same state(shaders, blend state, blah blah..) are rendered in one single drawcall.

So what we do to reduce the drawcall is called DrawCall Batching (clubbing together the draw calls)

Unity3D provides two types of batching —

Dynamic Batching — For small enough meshes, this transforms their vertices on the CPU groups many similar vertices together, and draws them all in one go.
Static Batching — Combines static (not moving) GameObjects into big meshes, and renders them in a faster way.

While discussing the drawcalls it is very important to discuss SetPass as well as it is a more expensive operation in a drawcall since it requires changing material and changing the render state. Which involves shader parameters like alpha blending, z-testing, etc.

Try to use one shader setup for different materials set up in a 3D object and enable batching

Key Take-Away for Developers —

How to set up for batching —

Different meshes should use the same material if not then make them use the same shader and use some batching tool to create texture atlases
Mark the meshes static this will enable static batching
There are APIs to combine mesh in runtime — https://docs.unity3d.com/ScriptReference/StaticBatchingUtility.Combine.html
There are some pre-requisites to achieve dynamic batching — less than 300 vertices, No mirror transform, single-pass shader

Note:-I read somewhere GPU instancing can also help to achieve batching, but I am too lazy to go deep into it. If you do read about it please leave a comment

There are optimization techniques for memory and coding practices which I will share in the next installment, i.e., Part 2.

References -

Credits —