Improving Loading Times in Talking Tom Hero Dash

TL;DR: By combining our experiences in game development, the team at Outfit7 were able to decrease the game load times in Talking Tom Hero Dash by 35% and improve the memory footprint, which also positively affected user retention. Players now reach the fun zone faster.

I’m Tomaž Treven, Senior Software Engineer at Outfit7. I joined this amazing team 10 years ago, around the time when the whole development team was working on My Talking Angela. Since then, I’ve been privileged enough to work with and learn from many incredible people on multiple projects at the company. During this time, I picked up a couple of things that might interest you and which we’ll explore in this post.

The Importance of Loading Times

In the quest to deliver the most fun to the largest possible number of people, we always try to tweak and optimize our games to minimize barriers and inconveniences and maximize fun experiences. We want players to be able to jump into the fun zone as easily and as fast as possible — and stay there for as long as they like. Part of how we achieve that is loading time, or how long it takes after installing the game to get to the part where the adventure starts.

Loading times are pretty vital. Back in the old days, Steve Jobs wasn’t satisfied with how long it took for the Macintosh to boot up. So he went to an engineer and encouraged him to speed things up. He believed that in a couple of years those machines would be used by at least five million people on a daily basis. If he could manage to reduce loading times by at least ten seconds, that would be around 300 million hours saved per year right there, which is at least 100 lifetimes a year. And it worked in the end — the engineer reduced the loading time by 28 seconds. Truly a great achievement and a great positive change for many, many people.

When looking at what we did with Talking Tom Hero Dash, I’m going to share what we did on the Android platform, since it’s easier to make iterative changes on there and we have a lot of low-performance devices (and, generally, all supported iOS releases run the game fine).

The first step in any performance optimization task is, of course, profiling or acquiring hard, cold, objective numbers in a controlled environment about certain characteristics. In our case, we wanted to find out which modules or processes were taking up so much time. And the benefit of getting those numbers would mean that, after any modification, we’d have exact numbers to tell us if the changes actually improved anything. And that’s important because sometimes we do something that we think will make things better, but it actually turns out worse! Like using a certain function, assuming it would be faster but then finding out it’s slower in our use case, or we spend a lot of effort optimizing a part that takes 10% of the whole loading time instead of focusing on the part that burns 90%. Like many wise developers have said, “Premature optimization is the root of all evil.”

Tools of the Trade

Unity out of the box provides some good tools that we can leverage for measuring/profiling performance. The most notable is the built-in profiler window. Normally we would profile on the actual device but, at the beginning, just to see where we were spending the most time, profiling in the editor was good enough. From the example below, we can see that we spent a decent amount of time loading huge amounts of data in the beginning.

At first glance, that code doesn’t do anything special that would cause such huge loading times. Sadly, enabling Deep Profile also didn’t give us the information on the exact data that was being loaded.

We were interested in memory, or what was loaded in memory to be precise and, for that, Unity offers another great tool — the Memory profiler.

When we started looking into it, what immediately became clear was that lots of assets were being unnecessarily loaded. For example, in the image above we can see that the general dialog atlas texture was being loaded, but in the beginning we don’t show any dialogs. Furthermore, we see that we have loaded textures, models, and animations for all of the characters, but we only need to have two. On top of that, all of the assets for all of the worlds in the world build and all obstacles in the runner worlds had been loaded. Suddenly the question of why the game was taking so long to load wasn’t such a mystery anymore.

Now “all” that was left was to figure out why everything was getting loaded — and fix it.

We’re probably all familiar with the MonoBehaviour class, the base for creating our own custom scripts that we can attach to game objects in order to give them various custom behaviors. Classes can, of course, have fields. The script below, for example, contains one number, which means that when the Unity engine creates a game object with this script attached, it’ll also reserve 4 bytes of memory to store that number.

using UnityEngine;

namespace Outfit7.Example {
  public class SimpleBehaviour : MonoBehaviour {
    public int Number = 123;
  }
}

We’re long gone from the old days when hardware was limited and every byte mattered. Take the legendary Super Mario Bros. game, created in 1985, for example, which was just 31KB. We don’t have to watch the size as much now, but that doesn’t mean you should load the game up and slow it down. In fact, today, even a simple small UI texture can be easily 10 to 100 times that size. So with that in mind, we also include fields of other types. Since we make games, we for sure want to include some images, animations, and even other GameObjects with their own assets.

As I’ve learned the hard way, whenever a particular GameObject is used, Unity will immediately load it and whatever is linked to it, even if a particular GameObject is not accessed or disabled. You’re probably seeing the problem now… In the image below, you can see that we have a ScriptableObject that has linked prefabs for all game states. Those prefabs include all sprite atlases, sounds, visual effects and other assets.

And this scriptable object is also included in MonoBehaviour on the first scene. By the same principle, all builder world assets, all characters, and all assets in the running levels are also being loaded into memory at the start of the game and are then always present in the memory, regardless of whether the assets are actually needed or not.

Armed with this insight, we prepared an action plan. We would switch from hard references to something else that would allow us to only load things that we actually needed for the state the player was in at that moment. And then we’d also unload those assets to keep the memory footprint as low as possible and to reduce out of memory exceptions. Usually it’s better to focus on just one thing at a time, but since both of those issues were so intertwined with resource handling, we thought we’d kinda tackle two birds with one stone.

Architectural Changes

First Attempt With Addressables

Our first attempt was with Unity’s preferred asset management packages — Addressables. Basically, the Addressable Asset System is a wrapper around their asset bundles. Before long though, we ran into a lot of problems with it.

The first was that by including that package, we’d already increased the game install size by a couple of MB, which was something we wanted to avoid. When you’re aiming to reach as many people as possible, every single thing starts to matter. From Google’s study, players are 30% more likely to download and install a 10MB application than one that’s 100MB, for example.

Another issue was that asset loading with that method took some time and, after a lot of trial and error, we couldn’t manage to get it to acceptable levels. Changing structure bundles, changing addressable bundle settings, changing loading thread priorities, changing time slicing… Nothing seemed to improve loading times as much as we wanted, especially for low-end devices (devices that were released 10 years ago, devices using a performance level of iPhone 5 or earlier, or recent devices developed for developing markets).

The documentation also wasn’t of much use and some new, weird crashes came with it. We didn’t have an option for synchronous loading that we thought would help. And we’d have to wait a bit for a newer version and then upgrade to the latest Unity version, which would further complicate things. Maybe we were using it wrong, but it just didn’t work well enough for us.

Second Idea With Custom Asset Bundles

The second option was to use asset bundles directly — the same approach we were using in our other, newer games. The issue with that approach was that it would require a lot of effort. In order to migrate properly, we would have to move all assets (including all shared textures, materials, shaders, and so on) to bundles in order to prevent duplicate resources in the game. For example, if we have a character with its mesh, textures, shaders, and such in one bundle, it’s okay. Whenever we’d need to show that character, we would have to have that bundle ready. But then you notice that the same shader is also being used on some random dialog. In that case, it would mean that the character bundle is now a dependency for that dialog bundle. This can be resolved by one of two approaches: moving the shared assets to a bundle that contains those shared assets; or by just including the asset in question into each bundle. What can happen in the first approach is that, if there isn’t a good hierarchical structure in place, all of the assets suddenly end up in a shared bundle. But the problem in the second approach is that we’d be increasing the install size.

Third Attempt With a Plain Old Resources Folder

Sorting out all of those cross-dependencies for the whole game was a bit out of our time budget. So we leaned into the next approach — moving assets to the Resources folder. The downside of that approach was that it takes a bit longer to build and load the game. But since, in the grand scheme of things, we don’t have as many assets as some big AAA games, that minus wasn’t that big for us. It also doesn’t allow you to change assets after release, which, again, wasn’t that much of a problem for us because we use other systems for downloading content after release, and whenever there’s a substantial game change, we always introduce it to the players as a new build. On the plus side, the Resources folder was simple and we were able to migrate each asset or group of assets to on-demand loading at a time.

In the below images, you can see that the catalog of states now only contains prefab names by which only the prefabs for the current scene are being loaded and then unloaded after leaving the state.

From:

SceneController sceneController = Instantiate(state.Data.SceneController);

To:

SceneController sceneController = Instantiate(Resources.Load<SceneController>("Scenes/" + state.Data.SceneControllerName));

Something similar was done for all of the runner assets. In the image below, you can see that all of the runner assets are directly linked to the runner catalog.

And after the change, each catalog entry contains just the path to the asset, instead of the asset itself.

Of course, it wasn’t smooth sailing the whole time. Some additional issues had to be handled. For example, on hub, you just see your selected character for the vast majority of time. But sometimes the big boss flies around, with another character caged on his plane, so managing the lifetime of such objects had to be implemented as well. Or, for another case in hub, there was an instance where you could only see the current and the next world, but the player could scroll over them. So we could leave it as it was, with all of the worlds loaded all the time, or loading/unloading just the ones you could actually see. Or we could use some hybrid system of loading, where we’d load just the ones you see and, based on some heuristic, load/unload others.

Results

After all of those changes, we observed that we’d managed to improve the loading times quite a lot — on average 35%. On one of the lowest performing devices, loading times went from roughly 37 to 29 seconds, and on one good device, it went from 10 to 6 seconds.

The memory footprint also decreased. It’s a bit hard to get good exact numbers on Android because applications are being loaded into virtual memories with shared libraries and so on, but generally, when entering the game, it was 10–30% lower if we were observing it in Android profiler.

Before:

After:

With the optimizations discussed above, the first day retention increased by 5% soon after release (and it might end up being even better with the adoption of the newest version). Plus, the amount of out of memory exceptions also decreased substantially. Not bad for a few people working on this over a couple of months! And it felt pretty good to reduce wait times and increase the fun for a couple (millions) of people.

Additional Resources

1. Ultimate guide to profiling Unity games: https://resources.unity.com/games/ultimate-guide-to-profiling-unity-games
2. Shrinking APKs, growing installs: https://medium.com/googleplaydev/shrinking-apks-growing-installs-5d3fcba23ce2
3. Assets, Resources and AssetBundles: https://learn.unity.com/tutorial/assets-resources-and-assetbundles