How to Fix LLDB: Couldn’t IRGen Expression (when using Carthage)

Timeline

Strava’s iOS & watchOS code is stored in a single repository. This repository contains all of our code as well as the compiled Carthage dependencies (Carthage/Build/iOS & Carthage/Build/watchOS). This is partly to reduce complexity for any given engineer (since they don’t need to worry about carthage bootstrap) and to avoid unnecessary building of our Carthage dependencies in our CI pipeline.

For months, Strava iOS engineers were unable to use LLDB from breakpoints in Swift code. The dreaded Couldn't IRGen Expression would be printed whenever po (or other commands) would be used from the Xcode LLDB prompt.

Although “workarounds” were found, all of them resulted in a drastically impaired debugging experience for code in (and out of) our application.

We were incredibly excited for the release of Xcode 12 when we saw the confirmation from Peter Steinberger in SR-12933 that his reported bug had been fixed.

At this point, we updated to Xcode 12, updated carthage, rebuilt all dependencies in our Cartfile hoping our issues were solved. But the problem was not fixed unless all of the Carthage dependencies were built locally for a given engineer. Building (carthage bootstrap or carthage build) Strava’s 20+ Carthage dependencies is slow enough that this wasn’t an ideal solution for each engineer to use long term.

This realization happened right before a Guild Week, which is a time where Strava routinely pauses all product work to focus on foundational infrastructure improvements/changes for the various platforms. We decided this problem had gone on long enough so it became a focus for our September 2020 Guild Week and (to give away the story) we solved it! We felt like all the examples around the web focused on how to solve it for framework producers (consumed via Carthage or otherwise) so we decided to publish how we’ve solved the issue as a framework consumer!

TL;DR

• Create an xcconfig file with the build options set to avoid serializing Swift debug information.

• Rebuild Carthage dependencies. Use the XCODE_XCCONFIG_FILE environment variable to pass in the previously created xcconfig file.

• Generate a series of file mapped UUID directories that are symlink’d to the location of the dSYM files in Carthage/Build/iOS and Carthage/Build/watchOS. Store these in Carthage/dsyms/uuids.
• Tell LLDB to use the file mapped UUID directories by setting com.apple.DebugSymbols[DBGFileMappedPaths] via a script build phase in some/all targets. This path should be absolute.

The Explained Solution

This section is broken up into two parts.

• The usage of -no-serialize-debugging-options to avoid having non-portable debugging information in the dSYM bundles that are a product of building the Carthage dependencies.
• The usage of the “File Mapped UUID Directories” feature in LLDB (on macOS) to communicate the optimal location to find the dSYM bundles for Carthage dependencies.

#1 — `-no-serialize-debugging-options`

From SR-12783 and SR-12933, we determined both that we needed to build our Carthage dependencies using the -no-serialize-debugging-options option of the Swift compiler and that this would likely work once we had updated all engineers to Xcode 12.

To set the build settings in the Carthage’s xcodebuild invocations we use a Carthage wrapper script. This sets the XCODE_XCCONFIG_FILE environment variable to get xcodebuild to use our xcconfig, which makes testing build setting changes easy. We transitioned this Carthage wrapper to a small example project with minimal Carthage dependency build time to replicate the debugging failures quickly.

To ensure this swiftc option was being properly passed from Carthage (via the xcconfig) to Xcode (and the other downstream tools), we used the log output provided when running a command like carthage bootstrap, shown below.

/var/folders/4y/7k0ckw7n69v9bgsvt6x7bnrc0000gm/T/carthage-xcodebuild.zz4u6R.lorg (from the command above) can be opened to find the xcodebuild output from each Carthage invocation. Looking at the options being sent to swiftc from the build system, we were able to check that -no-serialize-debugging-options was included in the options passed to swiftc when compiling the Swift sources. But, when doing a Carthage bootstrap using the following in an xcconfig, the issue was still not solved.

Inspecting the xcodebuild output using the above process, we noticed that while -no-serialize-debugging-options is being passed there’s also an unexpected -serialize-debugging-options option there as well.

Thanks to this post from Thi, we found out that Xcode will pass this value when the SWIFT_SERIALIZE_DEBUGGING_OPTIONS build setting is set to YES, which is the default. To solve this, we simple overrode the value of SWIFT_SERIALIZE_DEBUGGING_OPTIONS in our xcconfig file, which now looked like the following.

We confirmed that these xcconfig defined values are properly making it to xcodebuild by checking near the top of the log output.

And then, further down, we confirmed that the swiftc arguments now exclude the incidental -serialize-debugging-options.

After we confirmed the Swift sources are being properly compiled, we were able to validate on a virtual machine (with different paths to the project location) that LLDB was able to properly attach and symbolicate the codebase (Swift and Objective-C).

#2 — File Mapped UUID Directories

After confirming that the proper usage of -no-serialize-debugging-options worked in our virtual machine test bed, we transitioned to validating it on other internal machines with the real Strava project. To our surprise, the dreaded Couldn't IRGen Expression was still present. At this point, we figured we needed to better understand how LLDB was attempting to tie together the code running in our application to the dSYM files previously created for the Carthage dependencies.

We used LLDB’s image list to find the dSYM bundle LLDB was using for a given executable/dylib while the process was running (with a main interest in the Carthage dylibs). LLDB’s image list, in LLDB’s own words, allows us to “list current executable and dependent shared library images.” The output of this command looked like below while symobolication was failing in our example application.

This identified that the problem was a confusion between the available dSYM files on this particular engineer’s computer. In this case, there was one copy of the repo at /Developer/strava/iPhone and another at /Developer/strava/iPhone-copy. Both directories contained the built Carthage dependencies and the associated dSYMs. The project was being ran out of /Developer/strava/iPhone-copy but LLDB was choosing the other repository’s dSYM bundle, which was on a different commit without the -no-serialize-debugging-options fix from above.

We needed to explain how this choice was being made and therefore understand how LLDB finds the symbols for a given binary. Using [Adrian’s comment in SR-12933 plus LLDB’s own article about symbols on macOS, we determined a simplistic representation of the symbol search order looks like this:

1. Next to the executable/dylib.
2. Through any custom dSYM location mechanism.
3. Through Spotlight, using the UUID of the executable/dylib.

We weren’t copying the dSYMs into the application bundle (so 1 didn’t apply) nor intentionally using any custom dSYM locations (so ignored 2). To determine how Spotlight is capable of finding the dSYMs using the UUID of the executable we had to understand how to a) find the binary’s UUID and b) find the dSYMs associated with that UUID.

We were able to get the UUID for any Carthage binary using dwarfdump, based on this helpful StackOverflow post.

And, similarly, we were able to get some insight into how Spotlight would use the UUID for the simulator (`x86_64`, in our case) using mdfind, based on a different StackOverflow post.

This showed us that LLDB (via Spotlight) was finding dSYMs in multiple locations for the binary and choosing the one in a wrong directory. Simplistically, our task now was to either use a method that’s before Spotlight in the LLDB search path order or influence the Spotlight order. We were intrigued by the custom dSYM location mechanism, that occurs before Spotlight in the search order, so we investigated it more.

In LLDB’s symbols on macOS documentation, the file mapped UUID directories seemed like it had the possibility to be the most portable and wholistic approach. This required that we create a script to:

1. Iterate over all binaries in the Carthage/Build directory to get all the UUIDs.
2. Iterate over all these UUIDs and create the directory structure required.
3. Symlink the final node in the directory structure to the dSYM bundle for each UUID.
4. Set the com.apple.DebugSymbols[DBGFileMappedPaths] value in defaults.

To put these steps into more context with a concrete example, let’s walk through the creation of the file mapped UUID directory for the x86_64 slice of the Bluebird.framework dependency from above. We store these file mapped UUID directories in Carthage/dsyms/uuids and check them into our repository for ease.

LLDB, when debugging CarthageDebuggingTestsSmall, will now preempt the Spotlight search with this new Carthage/dsyms/uuids path to search for dSYMS using file mapped paths. Repeating the above for all Carthage binary UUIDs in a script (last command only needs to be ran once) ensures LLDB picks the dSYMs in Carthage/Build for all Carthage dependencies and solved our final issue!

To validate within LLDB (as a more obvious approach than just checking po), we once again turn to image list to ensure LLDB is using the expected path. Below, we can see that iPhone has been replaced with iPhone-copy, which is the desired path to the Carthage dSYMs.

To make this portable across our fleet of Macs, we run the final command from above (defaults write …) as a script build phase for the main target in the Strava project.

In general, com.apple.DebugSymbols[DBGFileMappedPaths] only needs to be set once but the script ensures that Strava repo duplicates and/or other (perhaps open source) projects that alter the the value in defaults are overwritten when the engineer is attempting to debug the Strava application in a particular repository.

Additionally, we’ve incorporated a repopulation of the file mapped UUID directories whenever a Carthage command is ran to ensure it’s always up to date with what’s in the Carthage/Build directory.

Closing and Thanks

• Thanks to Thi for his article.
• Thanks to Peter Steinberger for his article, SR-12933, and the continued updates on both.
• Thanks to Adrian Prantl (a compiler engineer at Apple) for his comments on both Peter’s issue above and SR-12783 and for the fixes in LLVM that both made LLDB errors more clear and potentially allowed our solution to work.
• Thanks to those who work on LLVM for the documentation about symbols on macOS.