Scaling iOS at Bumble: Part 2/3 — The Assessment

Recap

In our previous post we discussed how we came to be investigating project and build toolchain replacement solutions for our iOS codebase at Bumble Inc. We began executing proofs of concept for Swift Package Manager (SPM), Tuist and Bazel.

The Process

Our investigative process for each tool was performed in roughly the same manner:

1. Onboarding

With the exception of some light use of SPM as a third-party dependency manager, no-one on the team was fully fluent with any of the tools. This meant that at first, we needed to learn the basics, which could be achieved by creating a minimum-viable manifest for one of our node modules. This would involve learning the manifest format and core commands for building, testing etc.

2. Medium-sized proof of concept

Following on from the initial onboarding, we would then progress to converting parent modules, traversing the tree post-order. At each level we’d cover more of our core requirements and encounter edge-cases. Throughout, we noted learnings, limitations and fixes.

3. Full-app conversion

At this stage, since our module tree exhibits a strong level of uniformity, we’d now have gathered enough experience to fully convert the app’s child modules, and do so using automated tooling. How we achieved this will be discussed below.

Unsurprisingly, the top-level app target was its own challenge, and required considerably handcrafting. We only aimed to achieve a minimum-viable conversion at this stage, and didn’t expect tests to pass. That said, we’d usually have a fairly conformant conversion anyway, that would launch successfully to its home screen and usually plenty more.

4. Assessment

With a converted app target in hand, we would now have a candidate for assessment. But what did we want to know? Our analysis looked at these areas:

• Blockers: simple enough, any limitations or issues in the tool’s functionality whereby it was unable to serve our basic requirements
• Subjective user experience: how is the tool’s workflow in general? How is the experience working in Xcode? Can the tool be configured to get the kind of Developer quality-of-life that we want?
• Performance: data-driven assessments. How does the result actually perform? Are performance quality-of-life metrics acceptable? And most importantly, how are they affected as we artificially scale up the project size?

This last point deserves further discussion.

Simulating future growth

Tackling poor future performance scaling is at the core of this project. While it was not essential to improve current performance, it was absolutely critical to ensure any adopted solution would scale better than our projections indicated.

In order to do this we first identified a median-sized module. We then cloned it, once for its “legacy” handcrafted .xcodeproj, and once more with its converted manifest for each tool under assessment. We created varied sized batches of randomised-named clones and added these at key levels of our project graph. This allowed us to simulate feature growth in key areas, such as application features or shared platform features.

Needless to say, this particular step is highly dependent on the nature of our internal modularisation strategy.

The module converter

Unfortunately, this is not the point in the article where we announce our open source Xcode project conversion tool, which can magically convert projects into SPM, Bazel and/or Tuist!

While such a tool is feasible in theory, we were fortunate enough to be able to leverage our strong degree of existing uniformity in order to implement a tool that could spit out manifests for modules, given one of our key types:

• Logic modules, and their unit tests
• Resource modules
• Application modules and targets, their unit tests & UI tests

We were also able to leverage our existing XcodeProj-based tooling in order to parse the module graph. Then the uniformity of modules made fairly light work of processing globs of source files and resources. Build settings were almost entirely encapsulated in Xcconfigs. We mapped each module’s description into platform-agnostic DTOs, and used adapters which could output those to disk as Swift or Starlark source code manifests. SPM is the only of the three tools that makes this comparatively easy. However, our tool was able to borrow theSourceCodeFragment approach used here in order to write out Tuist’s Project.swift and Bazel’s BUILD files.

Investment into this tool was extremely beneficial: implementing an adapter for each new tool was not hard work, and allowed us to regenerate manifests for the entire module graph at the press of a button. We strongly recommend this approach to teams attempting this kind of migration!

The Results

So, with that preamble out of the way, we can get down to the good stuff!

📦 Swift Package Manager

As we’ve said, going into this project there was a certain hope that SPM would be the solution. The largest reason for this was a desire to remove tooling complexity — SPM comes packaged into the toolchain, is already integrated in Xcode, and fully supported by Apple. The iOS ecosystem has had a number of major third-party module/dependency management solutions over the years, and while these have been great enablers at the time, they also become significant points of failure. By relying on the first-party solution, the risk of breakage or abandonware ought to be reduced to near-zero. This is just one big reason why we had a desire for the SPM to come out the clear winner.

It’s important to restate that we wanted to use SPM in a non-typical manner: we wanted to leverage its local package dependency API to manage our hundreds of internal monorepo modules. Eventually, we’d also want to use it to replace Carthage for managing our small number of third-party modules, but it needed to perform primarily for the former use case.

Observations

SPM is certainly a well architected piece of software! Its library component, libSwiftPM, contains the implementation of all familiar swift package functionality. Linking this library into your own supporting tooling allows full customisation; want to implement a domain-specific swift package init? No problem.

On the other hand, although its source code is fully available on Github, the repo continues to state a 3-year old warning of instability. What’s worse, the library has poor or nonexistent documentation. Finding functionality is usually a case of searching the source code. The instability warning is certainly accurate, since during the project we noticed major refactoring between Xcode version updates. We’re happy that the tool is maturing, just as the language has. However, for the ecosystem’s core package management tool, this is also disappointing; instability rather undermines the advantages that it is supposed to have as the “first-party” solution.

There are also some interesting points to note in the separation between swift build’s core functionality and xcodebuild’s. Swift’s tooling is designed as a cross-platform solution, but iOS is very notoriously closed. While you may have success building low-level business logic and utility modules using swift build directly, eventually you will reach the point where xcodebuild is required. But fear not, SPM is fully integrated and it is possible to build packages directly:

$ swift package init - name MyTestPackage
$ xcodebuild \
  -scheme MyTestPackage \
  -destination 'generic/platform=iOS Simulator'

There are other subtle differences; binary targets are in fact an xcodebuild extension and not supported by core SPM at all.

App bundles

With that said, you should not be too shocked to discover that…

… SPM does not provide an iOS app bundle type!

So how do we ship a .app bundle managed by SPM? The answer is that even with SPM you can’t 100% abandon .xcodeproj; even if you can manage 99% of your modules as SPM packages, you will still need a top-level .xcodeproj that can package the result as a dot-app bundle.

The good news is that our proof of concept revealed that this top-level target can be extremely minimal:

Concept for minimal top-level SPM-based .app project

The above (crudely-edited!) image summarises what is possible: you can package the vast majority of your code and resources into an SPM module tree, but deliver .app iOS bundles by providing roughly the following in a plain Xcode project target:

• A “main” file, calling UIApplicationMain with the type of your UIApplicationDelegate
• LaunchScreen.storyboard
• App icon asset catalogue(s)
• The top level SPM module; the entry point to the SPM module tree. This can be added as a single local package dependency in Xcode

However, this goes against our intentions to un-track all .xcodeproj. To solve this, it’s possible to template this stub as well; for this we trialed XcodeGen, a lightweight project templating solution. For teams that have a large number of production targets, this stub itself could become a shared configuration.

So, with our top-level project needs covered in a pleasant way, we had no blockers and were quite pleased with the potential SPM offered us. But before we could decide anything we needed to process the numbers…

Performance

Concerns about performance were flagged early in our proof of concept. We first noticed Xcode’s Create build description build stage; our early build candidate would hang here for more that 200 seconds!

As you may recall, our existing project structure used explicit dependencies. This meant that each project had a link to the complete set of its dependencies. This also meant that our graph had a lot of redundant edges for transitive dependencies. Therefore by integrating a transitive reduction algorithm in our module converter tool, we were able to greatly reduce the graph size. However, even then Create build description was still taking in the region of 30–40s, for every build.

Having no clear options for locally addressing the issue, we opted to report the issue to Apple, and were fortunate to have it addressed in Xcode 14 beta 5. This reduced the duration down to an acceptable handful of seconds.

Having had a lot of focus on this particular issue, and having now had it resolved, the way forward appeared bright. Sadly this was still not to be the case! Moving to our data-driven performance analysis stage, we were forced to face the fact that other important performance issues remained.

Start with the positive…

With the Create build description issue fixed, build times were largely comparable between SPM-based projects and traditional .xcodeproj. SPM will always add a non-zero time for Package Resolution, however this is quite effectively cached. Uncached graph resolution can be rather slow, but the worst case is rarely encountered other than, for example, after an aggressive DerivedData clean.

However…

…the not-so good

Our first set of metrics look at a direct comparison of basic everyday performance:

The only metric under which SPM did not perform significantly worse than handcrafted .xcodeproj is in indexing. As noted above, we also lose performance in areas that were previously no-ops: Xcode file operations that are unnoticeable under .xcodeproj become double-digit second waits under SPM.

Our next set of data looks at Package Resolution time, scaling into the future. This is achieved using the module cloning technique discussed in the introduction:

Our 5-year module doubling benchmark is the x-axis half-way point; 500–750 additional modules. As you can see, cached graph resolution (red line) is quite effective. Uncached performance (blue) is slow, but not frequently encountered in practice…

HOWEVER!

This graph doesn’t tell the whole story: while cached graph (red line) resolution metrics may seem acceptable, the reality is that in practice these waits are encountered frequently!

Put another way: while you will rarely see 1 minute+ uncached graph resolution times, what you will often see is 10 second+ ~cached graph resolution times, as part of everyday tasks in Xcode

When integrating SPM into Xcode, the decision has clearly been made to create a user experience similar to Swift Playgrounds: the Developer works interactively, and Xcode eagerly updates state. Very ‘Apple’. The problem is, unlike in Swift Playgrounds, there is no manual override available. Instead, many everyday file operations will trigger graph resolution; folder content modifications (file create/rename/delete), but sometimes even file content changes!

As if the numbers weren’t bad enough, everyone who worked with the proof of concept project quickly noticed these sluggish performance characteristics

💃🏻 Tuist

In truth, we’d had some experience with Tuist prior to the SPM proof of concept. We were aware of its abilities, but had leaned heavily towards SPM. Now that we’d learnt SPM’s shortcomings, Tuist seemed much more appealing.

“It’s just a project generator”

Compared to SPM, Tuist is (effectively) not such a great departure from the traditional way of doing things. We have already mentioned Xcodegen, which is a very focussed project generation tool. Tuist is this also, but is more opinionated and builds a toolkit around this core functionality. However it is important to bear in mind that it is still “just” a project generator: you are still generating traditional Xcode projects, still building using xcodebuild etc. etc. Tuist is not a new underlying project format, it’s also not a new build toolchain.

Most importantly, Tuist modules are “just” Xcode projects. Therefore it is reasonable to expect similar modularisation-related performance characteristics. Having had bad experiences with module performance under SPM, this is certainly what we hoped to see.

Adapting our module converter tool to output Tuist manifests was quick work. In less than two weeks we were back at the stage of having a roughly converted Bumble.app.

Observations

When opening this first candidate project, it was apparent that our assumptions about performance had been correct; as with all other .xcodeproj, there is no dynamic graph, and so there is no resolution to be (repeatedly) performed. This strongly suggested we would not experience the dramatic regressions in quality-of-life metrics that we had under SPM. But what would our other analysis reveal?

Performance

The truth is that Tuist does manage a module graph. The difference is when graph resolution comes into play. Instead of constantly resolving the graph while working in Xcode, this operation is packed into Tuist’s core generate operation. This is the command which traverses each module’s Project.swift and spits out generated Xcode projects. As a result of this, it was the focus of the majority of our performance measurements.

In comparison to the poor SPM graph resolution experience, we were happy to find that, generally speaking, tuist generate performs well. As with SPM, Tuist caches manifest artefacts locally so there are the same cached/uncached characteristics. The graph below gives an outline of what we measured, again simulating future project growth by adding cloned modules:

Our analysis indicates that Tuist’s artefact caching performs well and that worst-case performance scales linearly. These are characteristics comparable to SPM’s, but with the critical difference that generate is always an explicit operation! It needs to be performed only after module structure changes, and is not run continuously as you (try to) work! This looked promising.

Aside: watch out for any internal security scanning services! As with building, tuist generate performs many small file operations. Without exclusions, security scanning software is likely to significantly degrade the performance of tuist generate. This is easily overlooked and can result in gathering inaccurate performance data.

Other metrics

In line with our expectations, we verified that quality-of-life metrics such as build, indexing and file operations performed equivalently in Tuist-generate vs. handcrafted Xcode projects. As such this data is not very interesting and we aren’t including it here. The only potential area of concern was project opening time:

This graph compares the worst-case (uncached) opening time between Tuist-generated projects (blue) and our handcrafted baseline projects (red). As you can see, opening time for the Tuist projects grows more quickly than for the baselines. Most likely this is a symptom of the change from explicit dependencies in the baseline to implicit (workspace) dependencies under Tuist; implicit dependencies require Xcode to do some graph resolution work, and this likely explains the difference.

Other features

As ever, it’s important not to forget that Tuist is more than “just” a project generator — it’s an automation suite for Xcode. As such it cannot break free from many pre-existing limitations. Tuist provides caching support, but it can only be implemented as a “helping hand” for Xcode’s own build artefact caching.

But there is much else that Tuist can do that might not catch as many headlines: did we mention it caches test results? Yes, it does! Not in Xcode, however, but using the command-line tuist test.

So, our investigations into Tuist proved to be far more positive. This left only one more solution to investigate. Our tooling was in a good place to turn it towards another manifest output format, so it shouldn’t be much work, right?…

👹 Bazel

…also know as “The Beast”.

Observations

Bazel has been rapidly growing in popularity and stands apart from SPM and Tuist. The former are Apple/Swift/iOS ecosystem tools which are, with some minor exceptions, wholly developed for the needs of a particular set of Developers. Bazel is most definitely not this. Its website describes it as for “your multi-language, multi-platform projects”. It is fundamentally a general-purpose tool.

This means that you definitely do not get out-of-the-box support for the needs of building a typical iOS application. In fact, out-of-the-box you get no ability to build at all! Bazel itself is an orchestrator which must be combined with a set of rules for the tasks you actually want to perform. So, right away, you’re presented with Bazel’s first challenge; which rules to use? Starting from a web search, you’ll likely find:

• https://github.com/bazel-ios/rules_ios
• https://github.com/bazel-ios/xchammer
• https://github.com/bazel-ios/rules_apple
• https://github.com/bazel-ios/apple_support
• https://github.com/bazel-ios/rules_swift

etc. 😕

If, more likely than not, your team lacks anyone with previous Bazel experience, then this selection process can be bewildering. Not to mention that these rules are versioned, subject to breaking changes, and often not comprehensively documented. What’s more, you now have dependencies inside your dependencies, because Bazel rules can (and do) depend on other Bazel rules 🤯.

When working with SPM and Tuist we took the logical bottom-up approach — start by converting the smallest node modules, then work upwards to the app-level target. This works both practically, and from the standpoint of onboarding a new tool; you can learn the tool, simplest modules first.

The problem with Bazel is: without previous experience, there’s a good chance you won’t be able to select the ideal rule sets that support every one of your use cases. What will happen is you’ll progress happily with your proof of concept until you hit the brick wall of an unsupported use case, and are forced to begin again from (possibly) scratch. This is most likely to happen once you reach the app layer; for example, rules_ios exports the xcodeproj() rule. This looks good at first, but proves to be underpowered for the needs of a serious production app.

So here’s our top tip: as of 2023, the gold standard rule set for Xcode projects is rules_xcodeproj. It has support from other similar-scale teams such as Spotify and Reddit, and has recently been taken under the banner of Mobile Native Foundation

So, with rules_xcodeproj in your WORKSPACE file, it’s time to convert your modules. Whilst having the de facto best rule set selection is an important start, Bazel is still a steep learning curve! Its manifest language, Starlark is like a “Python-for-beginners”. Tip #2, don’t make the mistake of sketching some regular Python3 and pasting it into your BUILD file! Starlark is a limited subset of Python’s functionality.

Next, you’ll need to find out how to serve your build use cases. This will be fun, since documentation of rules can be patchy. You’ll find yourself digging into the rule source code quite often. Here is a short list of some of the specific challenges we faced:

• We use a decent number of precompiled framework dependencies; finding a Right Way to vend these took a few (mis-)attempts
• We make heavy use of xcconfigs. At their heart, xcconfigs are maps of build settings, with some features for supporting #includes, overrides etc. As far as Bazel is concerned, they’re just maps of build settings; there are no rules that let you consume xcconfigs as files, preserving the inheritance features provided through Xcode. So, for our PoC we did a minimum-viable export of some functional settings, but we would need much more for production

Tip #3: as soon as possible, begin creating macros that cover your use cases. These will be domain-dependent of course; whatever common types of business logic, test types, demos, playground-style workspaces etc. Enumerate them and write shared macros for each.

But we’re not done yet: you’ll also need to find ways to debug your builds. Yes, you read that right — you’ll need to find ways to debug your builds! SPM and Tuist are tools built for your use case. Bazel is not. That means that when a build fails you can’t expect it to give you very specific information — you’ll have to find that yourself! Some information that may get you started can be found in a small number of blog posts, but we predict that any production deployment will require a solid development environment with support for debugging Starlark BUILD files. There are extensions available, such as this one for VSCode, but we had some challenges getting these to work reliably during our proof of concept.

Results

Although some of the above is quite negative, the point is that Bazel has a steep learning curve. The amount of time needed to convert our proof of concept app-target for Bazel was significantly longer than for SPM or for Tuist. We had a module converter that could be repurposed to output Bazel BUILD files, but it took much longer than expected to learn the lessons needed to develop macros for our use cases. There was frustration and false starts. However, once things begin coming together you start to see where Bazel shines; it’s fast, and even without measurement, caching effects are perceptible.

Builds

So where’s the chart demonstrating how much faster it was? Well, that is data that is rather hard to produce. Since the majority of benefit is gained by shared caching, we would need a trial deployment in order to gather meaningful data. In our case, clear choices were made that cut off our investigations before this point.

Xcode

The iOS developer experience centres around Xcode, so having a quality experience within it is critical. This is not a concern with SPM and Tuist, two tools built for the ecosystem, but Bazel is not an iOS development tool. So, a high-quality experience in Xcode cannot be taken for granted. Fortunately, however, rules_xcodeproj delivers the goods. While we didn’t go so far as to completely recreate the functionality of our existing projects, we did need to go deep enough to understand if the rule set could provide enough power and flexibility. We believe we saw enough to have this assurance.

That said, the resulting projects are rather unfamiliar. It’s possible to use either bazel build or regular xcodebuild from within Xcode. The former will take some getting used to for existing Developers; it certainly is a jarring difference from Xcode’s native experience. Compared to either Tuist or SPM, Bazel will be a more initially disruptive choice.

But there is more to like as well — Bazel includes a powerful graph querying language and some interesting supporting tooling such as bazel-diff, which would replace more custom tooling we’ve implemented internally.

So — at least once set up — Bazel appears to deliver on its hype. But was that enough to convince us to invest in it?

What’s next

In this post we summed up our experiences evaluating these three tools as possible solutions for our growing pains. Tune in to our next post where we finally reveal our decision, our reasoning, and what happened when we made the switch.