Step by step tracking down a macOS Beta regression

This morning a report came in that Xamarin.Mac Classic applications were broken on High Sierra (currently in Beta).

This was somewhat surprising, as our automated introspection tests were passing last time I checked. Thinking about it though, Beta 3 had recently dropped and we are still in the process of filing radars / fixing issues on our end, so it was rather possible.

I figured it would be a great investigation to write up.

A quick test on my laptop confirmed the crash:

2017–07–12 10:32:02.849 Classic[13505:303375] ### Failed to Soft Linked: /System/Library/Frameworks/Contacts.framework/Contacts
#5: 0xa782502b libsystem_platform.dylib`_sigtramp + 43 
#6: 0xa3884183 GeoServices`initCNPostalAddressStreetKey() + 48
#7: 0xa3911a8c GeoServices` + 54 #8: 0x0012b502 dyld`___ZN16ImageLoaderMachO18doModInitFunctionsERKN11ImageLoader11LinkContextE_block_invoke + 34
#9: 0x001151ed dyld`dyld3::kdebug_trace_dyld_duration(unsigned int, unsigned long long, unsigned long long, void () block_pointer) + 210 #10: 0x0012b450 dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 422

I then wondered if this was Classic specific (likely our bug) or 32-bit specific (likely Apple’s bug). A quick test app confirmed that any Xamarin.Mac application in 32-bit crashed.

Looking at the stack trace, the following story is told:

  • We’re in the image loader for MachO, so we’re loading some native frameworks, very likely Apple’s
  • doModInitFunctions sounds like static initializer, and ___ZN16ImageLoaderMachO18doModInitFunctionsERKN11ImageLoader11LinkContextE_block_invoke agrees, so we’re handling native static initializers
  • And that code is calling initCNPostalAddressStreetKey. CN is a likely prefix for Contacts, which explains the error about loading Contacts.framework/Contacts
  • file /System/Library/Frameworks/Contacts.framework/Contacts confirms that the library is 64-bit only
  • I was not aware of GeoServices, but it sounded like a private framework and I found it in /System/Library/PrivateFrameworks

My working theory became — “There must be some library we are binding, that depends on this private framework, and that private framework is not respecting 32-bit restrictions”. Neither use case tested uses the linker, so it could be any API we bind.

Bug reports (Radar) to Apple always go better if you can eliminate all managed code and provide a simple objective-c app, so that my was next goal.

To do that, I need to track down a publicly available framework that uses GeoServices, use it in a 32-bit obj-c app, and hope it explodes. Since I knew nothing about GeoServices, I wrote a tiny script[1] to dump the framework dependencies to track one down. CoreLocation was a great candidate, so I created a new obj-c Cocoa app, added a references and ran. That didn’t crash, but maybe a reference isn’t enough. I added an invocation:

CLLocationManager * mgr = [[CLLocationManager alloc] init];

and that crashed exactly the same as my Xamarin.Mac app.

I then had a co-worker who was on Beta 2 try that, and it worked. Which means it was a likely Beta 2 to Beta 3 regression, which I filed here.

I’d like to wrap up with a request that Xamarin.Mac users prioritize moving their applications to 64-bit Unified. This is not the first 32-bit regression we’ve seen in High Sierra, and Apple announced that it is the last macOS that will run 32-bit applications “without compromise”.

[1] — An aside, this could have been done with a unix one liner if my unix skills were more polished, but I knew I could bang it out in C# more quickly.