Ollie’s App Architecture Continued

Published in

Ollie

7 min readFeb 14, 2024

This is a continuation of our last article about our app architecture and our reasoning behind being a native app vs. a cross platform one.

Local Data State Model

Our app is more data model & business logic than view logic. Due to the reactive nature of SwiftUI, we made it a bit too reactive, which we plan to move away from. But the basic data model types are:

Photo: A singular photo, from multiple providers or ‘destinations’, be it Google Photos or PhotoKit. Has many attributes such as location, AI image tags, source, time, album membership, etc.
Album: This represents “interesting” sets of photos. It could be all the photos in a month, a photos of certain category (e.g. pets), or a certain trip. This is somewhat replicating functionality from Google Photos and Apple Photos but we needed to do that in order to help users sort photos with a certain theme.
PhotoStack: A small subset of photos from an Album that is not too overwhelming to sort or train an AI model with. Usually 15 to 30 photos, which by most people, can be sorted through in less than a minute.

All of these basic models and their attributes are persisted in our sqlite3 DB, wrapped by GRDB, which is then wrapped by our own Repository & Query pattern that generates and persists these model objects.

We also generate some Apple albums as a backup source of truth & form of export. All photos you mark as favorite, trash or ‘keep’ in our app is also put into an equivalent Apple album inside the Apple Photos app with PhotoKit. So even if you delete our app and its database, we can restore most of your app state via these Apple albums.

Event Generators & On Demand Actions

We then generate model objects to populate the database and albums via various event generators and on demand actions:

Event Generator: PhotoKit Import Watcher

Most photos come in via an import action every time you launch the app, which queries photokit for the metadata of your photos to kick start various other services and put them in our DB. It also detects when photos are deleted to mark in our DB as ‘deleted’.
Album Calculator: Since this is a pretty quick operation, this doesn’t need to be done as a background service action and is just done on demand for new photo imports
Stack Generator: From the raw stream of albums, we then generate organization ‘stacks’ for the PhotoStackLoader to use within those albums.

Photo Processor Service:

This is a ‘background’ service that does the heavyweight raw image → vision tag scanning and trip album calculations to put in the DB to make other actions act quickly. It acts continuously in the background until it is done scanning your entire photo library, watching the DB photo stream.
Tag Extractor: Associates tags for each photo, such as ‘beach’ or ‘flower’.
Trip Calculator: Takes photos and puts them into various trip albums or generates new ones for them.

We even have some fun progress views in our developer panel.

On Demand: PhotoStackLoader

This is used whenever a view needs to load a stack of photos for the user to organize. It’s fed a PhotoStack object to load.
Loads the full image thumbnail + photo from PhotoKit for all photos in the stack.
Generates the AI model rating for each photo via on device inference. We do this on demand since the model changes every time a PhotoStack is committed, so we can’t do it in the background photo service.
Calculates related / similar photos, called a ‘bracket’, via a 2 minute time window, detects if they are similar or not and chooses the best photo out of them. A stack can have multiple brackets within it.

On Demand: Post Stack Committer

After a user finishes a batch, we need to save and process the results.
Commits user’s decisions to the DB
Trains AI model with new photo data and decisions
Modifies Apple albums with new state (new favorites, etc)
Deletes photos marked as trash by user with a PhotoKit prompt

Subscribing Event Watcher: Gphotos Uploader

Any new ‘favorite’ images, if the user has enabled it, gets uploaded to google photos also via the GPhotos library.

Major Dispatch Queues

We found that with Swift Concurrency’s single shared concurrent queue, putting all network, PhotoKit, CoreML, vision calls as basic async/await functions to be an exercise in creating heisenbugs and unobservable deadlocking. To mitigate this we explicitly created a set of dispatch queues for each major subsystem that has a heavy workload:

Photo model object update queue
GPhotos Upload & Queries
Apple Vision Calls
PhotoKit Queries and Photo Loads (This includes adding / removing things from Apple albums)
Database Actions
CoreML model inference & updates (this is what our custom model runs on)
Logging systems that already came with their own queues

Each queue was an exercise in hoping we didn’t need to do it with Swift Concurrency / Combine, but ending up needing to in the end to avoid deadlock and heisenbug issues. More details about this in the next blog post!

View Architecture

Like most SwiftUI apps, we are side effect state driven. Flip a state variable somewhere, and your view changes in reaction to that state change as a side effect with observer bindings! We recently changed screen management to be purely UIKit driven, but in the past when we were SwfitUI driven we had top level screens being controlled by several state variables within the top level App & AppState objects.

Top level views were swapped in to a root view:

Login
Onboarding
Main

We had an overlayView as a special global view that any view model can set over the rootView, as an alternative form of .fullScreenCover() that gets around some bugs and limitations in that API.

View Models

Many full screens have their own associated view model which executes a lot of business logic on their own. A lot of PhotoStackLoader actually occurs in the StackLoadViewModel, Temporal stack generation happens within the TemporalStackViewModel, interacting with various managers and so on. A lot of services get generated as part of DI initialization and get started by app launched or app foregrounded notifications from the app to manage the lifecycle properly.

This heavy ViewModel architecture is something that doesn’t jive well with SwiftUI in some fairly subtle ways and is something we eventually plan to move away from. Also SwiftUI’s performance for large image collection views is hard to make performant compared to the equivalent UICollectionView we have found, since it is really easy to trigger SwiftUI to act in a non lazy loading way compared to UIKit

Navigation Stack Manipulation Alternatives

The StackLoadView is kind of a goofy hack, with the loading view being the parent view of the grid view / carousel detail view navigation stack. This was mostly put in because you don’t have the fine grained control that you have in UIKit that allows you to manipulate the navigation stack while also being compatible with iOS 15. We wanted to be able to load a new stack once an old stack was done without going back to the home tab and without being able to ‘go back’ to the old stack essentially. In UIKit this isn’t a hard thing to do.

Similarly with our login → onboarding → grid → main tab navigation, we had to implement it as a root view swap + overlay because you similarly cannot manipulate the navigation stack in iOS 15 that well to prevent going back to previous views or other complicated transitions. This has created a lot of hacky complexity in our transitions that has been the source of several major bugs that we got rid of with transitioning our screen management to UIKit.

In the next articles, we will go into the lessons learned from using Swift Concurrency and SwiftUI in a photo heavy, multithreading heavy application, the surprising footguns you can run into with using them. In another article we will go into why we decided to migrate to UIKit & Dispatch to solve them quickly compared to radically changing our data model architecture.

This is part of a series of blog posts where we go into: