How Tokopedia Achieved 1000% Faster iOS Build Time
Huge iOS App build takes forever is a myth! See how we reduced our build time by an astounding number that nobody can believe!
One of the greatest things about working at Tokopedia as an engineer is the opportunity to improve the experience of our Toppers (cute name to call Tokopedia users). Also, it’s equally rewarding to create an environment that makes us work faster and efficiently.
However, building a great experience is much easier said than done, particularly when building a project for iOS. The time we spent to compile the apps from the source code -or technically known as Build Time-, was UH-MA-ZING. It normally took 50 minutes to an hour and a half for the build to complete.
For that reason, we’re constantly looking for ways to decrease the amount of work that we do on our builds. And this time, we made it! Check this out!
The Background
All good stories should have a background, isn’t it? Well, here we go, this story began a long time ago. As a team, we have been going through a lot of ups and downs. We once experienced 50 minutes clean build time and changing just 1 line can take another 5–10 minutes to build. Moving from our rock-bottom state, we started our quest of improving development experience.
We’ve gone through a lot of valleys and milestones with one objective in mind: ‘How can we improve our developer experience?’. We have implemented a multi-module strategy which means now we only compile what module that needs to compile. No need to compile other modules that don’t need to recompile anymore! Yes! The first victory that enables developers to do their magic faster!
Another win was when we moved our CI system into the in-house CI system. We had struggled and paid A LOT to cloud CI providers but the build time started to pile up. Our main suspect was due to a low spec machine being provided to serve us but upgrading to a pro machine cost even more. It was a balance that we preferred not to take at all.
The Problems
“The build process in CI machines is too long”. Even when we’re using ‘trash can’ and iMac as our Jenkins executor, it still took about 30–50 minutes for the build to complete. Yep, you can feel the agony. And what if you just forget to change that 1 pixel margin? Yes, another 30–50 minutes then. Don’t forget you have to wait in the queue once in a while. Dude, now we have a lot of coffee to sip on. ☕️
“Queue is longer as we grow in size!”. Our developers are working on at least 1 feature at a time. Sometimes they can even work on 5 features at a time. Now, you do the math, with our 40 developers, how many builds needed per day? Now I pity the poor Jenkins machines.
“On Monday, it feels like hell!”. We adopt a weekly release paradigm. We want to push the feature as fast as possible, give the product team a platform to test and improve our end-user experience.
As you can see, the flow ends and starts on Monday and while there are some features that are able to finish on Friday, most features take their time to test more thoroughly and get merged on Monday. That means a long queue in Jenkins. If you then have a conflict? Resolve and then start from the end of the queue again.
“Even in local machine, clean build takes way too long”. Sometimes we have to do clean builds, aren’t we? Even on a local machine that only executes 1 thing (build the project) it still takes up to 20 minutes! 🤯🤯🤯🤯
Too much pain isn’t it? All the previous journeys have enabled us to take greater control of how our build is done. We now have the full capability (and responsibility) to do more and thus improve day to day development experience (this is -at least- what we believe we’re doing). Now we come to the point where we are trying to substitute our build system and improve our build time.
The Solution
Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. It uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms.
Why Bazel?
We choose Bazel as our painkiller because it is Fast & Reliable. Bazel only builds what is necessary. It always produces the same artifact as long as you don’t change the input file. Another superpower that xcodebuild doesn’t have is Remote Caching.
Remote caching basically means Bazel caches every build output in a remote server and uses them as needed. That means no rebuilding kinds of stuff that doesn’t change. Just use cache 😎
Bazel is also Extensible. It is using Starlark language under the hood. That language is built on top of functional programming, so you can create your extension to create well, pretty much everything. Unlike xcodebuild, Bazel’s syntax is declarative and human-readable so we can seamlessly know the process of how Bazel will build our lovely Apps.
Now, all of sudden Bazel becomes the pretty obvious answer. But, applying them to our pretty messed up codebase is nowhere near easy.
The Implementation
By design, Bazel does not really support building mixed-language iOS projects but by learning how Xcode works when we build the project, we can mimic the process and translate it into Bazel rules.
First, Xcode will copy all of your objective-c headers into the headers directory. In this phase, we can use objc_library rule with all of your header files as source.
After doing that copy stuff, Xcode will continue to compile all of your Swift files to produce the swift bridging header and the swiftmodule files. We can use the swift_library rule to produce the same things as Xcode did.
Lastly, Xcode will compile our objective-c source to produce the modulemap and the executable files. To reproduce this step we will again use objc_library, and after that long step, we are able to build a multi-language app using Bazel. Now it sounds easy enough, but now you try to bet on how long it took us!
The Hurdles
Like all complex apps out there, our app uses many third-party frameworks and it is built on top of various languages such as objective-c, objective-c++, c, and of course swift. It’s so awful when we know that we must configure build rules for each of them to be able to build on Bazel.
So after doing some surfing on the internet, we found some tools created by Pinterest called PodToBuild. It is a tool that helps you convert your Cocoapods dependencies into Bazel build rules. However, that tool even isn’t adequate for our multi-language dependencies 😞.
By nature, we are too lazy to configure build rules for each of our third-party frameworks. So we push our creativity to make our own build rule generator, which basically will read all of the dependencies configurations and determine whether they are swift library, objective-c library, or multi-language library and create build rules for them. Thank god, this kind of approach eliminates the first approach when we must manually configure build rules for our many, many third-party frameworks.
The next obstacle is how we can orchestrate our multi-language build even though Bazel isn’t officially supporting this. The problem appears when some swift module wants to import the objective-c module and vice versa. It’s almost made us give up and stop dreaming that we can migrate our build system to Bazel but thank goodness that we found a workaround. The workaround is that we fork the Bazel repository and do some modification on the code. We know this isn’t a good way to solve our problem, but this problem will cease to exists when we’ve completely migrated into swift.
Last but not least, build time optimization seems to be a challenge for us, because after we migrated to Bazel, the build time still isn’t satisfying. The build time is increasing by over 25% instead of reducing ☹️. To get a piece of advice, we headed to Bazel Slack, and we got an answer from the members of #ios channel. It turned out that the problem occurred because the sandbox feature was enabled which caused the build time to increase.
The sandbox feature is what makes a build very safe and hermetic but after some consideration, we don’t need it right now. To disable the sandbox and optimize the build time we added flag — spawn_strategy=local and — features=swift.use_global_module_cache, and after adding those flags, our build time gave us a satisfying smile 🙂.
The Result
After using Bazel to build our app, we are able to see a dramatic increase of an average 500% faster than using conventional building methods in our CI builds. Once, we could watch an episode of Game Of Thrones, waiting for a build to finish. Now we can be more productive! And our overall workflow and productivity also soar into the sky.
And you know what? This isn’t even the final form! Bazel provides caching capability so that we can reuse our previous build output for the latest build! Another transformation, eh?
This is our build duration using remote caching:
As you can see, this causes an increase of an average 200% faster than before, making the total 1000% faster than conventional building methods.
Lesson learned
After all the tiring efforts, finally, we can build our codebase with Bazel and sip the sweetness of using this build system. We knew Bazel wasn’t designed to build mixed-language iOS projects and we decided to fork and customize Bazel to meet our needs. A fantastic lesson that we learned is getting to know about the build process behind each build run in the XCode. The XCode build logs seem like a place full of treasure. Learning the process behind XCode build is the reason for us to be able to build a mixed-language codebase with Bazel.
However, because we know that this is not the best solution for the long run, we have to keep improving. In the end, we still have a long way to go! As Albert Einstein reminded us “Once you stop learning, you start dying”. This also underlies one of our company’s visions “Make it Happen, and Make it Better”.
The migration process is quite tough, we have to create a dedicated team, and also it takes around 6 months in order for this migration to be successfully used for a production build. In short, it is costly and takes time. But seeing where we are now and our newborn baby, it is a very satisfying experience. And the most fascinating thing is we can boost our performance as a team!
Thanks to Bondan Eko Prasetyo for co-authoring this story with me and becoming the spearhead in Tokopedia iOS x Bazel.
