Introduction to Yap Database
A few months ago, my team at work decided to use the Yap Database framework in our new iOS app. I had never heard of the framework before, so I started looking through the documentation to get familiarized with it. At first, the documentation was initimidating. It was dense with new terminology like views, mappings, connections, transactions, and grouping and sorting blocks. The framework seemed inscutable at first glance. However, after just a few days of using Yap, I realized how enormously powerful and intuitive the technology is once you overcome the initial learning curve. My hope is that this post will help other developers do just that.
If you would like to follow along in XCode, the full project can be found here.
What is Yap?
Yap is a database framework built atop SQLite for iOS and Mac applications.
In everyday terms, how does it work?
Before diving into the details of Yap, let’s consider an example that will make Yap far more familiar. If you want to jump directly into the details of Yap, bear with me for a second, as understanding Yap at a high level will make the next sections more-easily digestable.
Imagine there is a particularly dedicated high school librarian named Mrs. W that wants to optimize how quickly teachers can find the books they need. This library is a bit different than most because only Mrs. W can pull books from the shelves. To access the books, teachers must send Mrs. W a list of the books they need in advance of their visit. These requests can be for one or many books, and can also be broad requests like “I’d like all of the books written between 1910 and 1915.” Once she receives one of these requests, Mrs. W compiles a list of all the books that meet the provided criteria and where each book can be found in the library.
So how does Mrs. W know where each book can be found? She decides to sort the entire library first by genre, and second by each book’s uniquely-identifying ISBN number. By knowing these two pieces of information, Mrs. W can know exactly where to look for any given book she needs because no two books can have the same genre, ISBN pair.
As an added convenience, she optionally allows teachers to include a sorting order in their request. Thus, a teacher could request, “I’d like all of the books written between 1910 and 1915 in ascending order by the author’s last name.” Rather than enforcing one strict sorting criteria, Mrs. W can now sort each compiled list individually and provide each teacher with exactly what they need.
To further optimize her library, Mrs. W keeps these sorted lists in case a teacher ever needs access to the same set of books. If they do, she knows exactly where to look for each book and how the group should be ordered. As an added bonus, she also evaluates every new book that is added to the library to see whether it meets the criteria of any of the lists she has curated. If so, she’ll add it to the list and properly sort it. Additionally, if a book is removed from the library, she removes it from all the lists it belongs to.
This precise system of organizing and accessing data is, at a high level, the beauty of Yap Database.
How is the database structured?
The main database for Yap is structured as a collection-key-value store. Collections provide an extra level of organization on top of the typical key-value database. Using the library example, a sensible collection value would be genre. Thus, the books would be separated into their respective genres, and then the books could be sorted by the unique key (ISBN) within each genre. In this case, the value would be a Book object, containing its title, author, and other details. It is worth noting that within each collection, each key must be unique, thus every object can be uniquely identified by its (collection,key) pair.
How do we access the database?
We access an instance of Yap Database, which is only intantiated once in an application, through connections. Establishing a connection with the database opens up a line of communication between the application logic and the database. Once a connection is established with the database, the application can send read and write operations, known as transactions, along the connection and receive responses, like a call made over a telephone wire. (Hint: Think connections are persistant, while transactions are momentary.)
This is how we initialize the database and a new connection.
let database = YAPDatabase(path: databasePath) connection = Database.newconnection()
Yap Database supports concurrency so we can have multiple read requests occuring at once over different connections. Read-writes are put into a queue and executed one at a time, and can happen concurrently with read actions.
Connections are particularly powerful because they each have their own cache. Thus, when we request an object over a connection, the deserialized object will be added to its cache. That way, if we request that same object again over the same connection, there will be a double performance boost because the object is already cached and deserialized.
Each connection’s cache is kept up to date through the use of snapshots. Snapshots are similar to git commit numbers, and each write action to the database increments the database’s snapshot number. When a connection is first created, its snapshot number is copied from the database. However, after a write transaction to the database, the connection’s snapshot number is then outdated compared to the database’s, so it ‘pulls’ the database changes (known as a change set), updates itself, and increments its own snapshot number to match the database’s. This is how each connection’s cache stays up to date with the database.
What’s cool about it?
The best part about YAP is its use of extensions. Extensions are a way of accessing and maintaining particular subsets of data. There are a few different kinds of extensions. In this tutorial, we’re going to focus on the most common type: views.
Views are a way of viewing a particular subset of data. In the library example, views are the book lists that Mrs. W compiles for each teacher. If a teacher had asked for all of the books written by Walter Isaacson and Charles Dickens, this is what that view might look like. Each view is made up of one or more groups. There are two groups in the following image, one for Dickens and one for Isaacson. Within each group, we can find a list of collection, key tuples that represent the works of each author. These tuple values don’t actually hold the books themselves, but rather a pointer to where those objects can be found in the main database. (Remember: every key in a collection is unique, so no two objects can have the same collection/key pair!)
Now, assume we are building an app for the library and need to present a tableView with the books sorted alphabetically by title. Thus, the view is going to have a group for each letter of the alphabet, and each group will contain a sorted list of every book that starts with that letter. This is a truncated version of what we want our view to look like. (Note: The collection / key values here were chosen randomly.)
So how do we create these views? The following block of code is all that we need (notice we can use regular objective-c/swift code!).
Don’t be nervous if that syntax seems weird at first, we’ll go over each piece individually.
Let’s break down each of these blocks one at a time. First, there is a grouping block, which determines which objects should be included in the view.
The important thing to note is that this method will be run on every object in the database when the view is initialized and upon every object that is added or modified by a write transaction. From the libary example, this is analogous to when Mrs. W first compiles the list of books and when she examines each new addition to the library to see whether it meets the criteria of any existing list.
Now that we have the idea behind the grouping block, let’s look at the code itself. First, notice the omitted parameters in the arguments list (i.e. the “_” spaces for argument names). These omitted parameters provide more information about the object being passed into the block, like its collection and key values. However, we typically inspect just the object to determine whether it should be included in the view. (More information can be found here)
Once we determine whether the object should be included in the view, we return the name of the group we want to assign the object to, as a String. In this case, the guard statements validate our input. Before adding the object to a group, we need to make sure it is a book. If it fails this validation, we will return nil and the object won’t be assigned to any lists. If it passes, we return the first character of the title. Why this value? Remember that we are returning the name of the group we want to assign the object to, so we would return “A” for A Tale of Two Cities.
The sorting block is the second component to creating a view. From the library example, this is the equivalent to Mrs. W asking how to sort the compiled list of books. The sorting method is very similar to a typical compare method: the arguments are two arbitrary objects from the group and the return type is an NSComparisonResult of how we would like them to be sorted. Once again, this logic is run when the list is initially compiled and upon each new write transaction to the database, so the list is guaranteed to always be sorted correctly.
How does the view we created translate data to a tableView?
Simple: we use a nifty tool called mappings. If you consider the image above, our view has already been populated and sorted, so our tableView has pretty much been constructed for us. Now, we just need to map each group to its associated UITableView section. (Note: this would work equally well for collectionViews.)
There are two strategies we could use to map each group to its associated tableView section. First, we could declare the mappings and their order statically.
However, this strategy is rather brittle, especially because we may not have a view for every letter in the alphabet if there aren’t any books that begin with that letter. Instead, there is a far better strategy, which is to declare the mappings dynamically. In essence, we want to find a way to tell Yap which groups to include in the mappings and how the mappings should be sorted. Sound familiar? This is exactly what we used grouping and sorting blocks for earlier when we needed to determine what should be included in our views. We use the same strategy here to determine how we should map our groups to tableView sections.
There are two other things to note about this mappings method. First, it should always be called directly after beginning what is called a LongLivedReadTransaction. This effectively freezes the database in time while we are performing the operation we want. Secondly, we must use the updateWithTransaction method in a transaction block in order to initialize the mappings object.
These mappings become particularly useful when we are writing the tableView’s delegate methods because the mappings object contains important information about the view.
This, at a high level, is the power of mappings. Views would not be useful on their own without the ability to quickly translate their contents to tableViews and collectionViews. By combining views and mappings, we can preset how our data should be organized and presented, thus drastically reducing loading times. More information on mappings can be found here.
Persistence
Like Mrs. W’s lists, views persist over time, so they only need to be initialized once and then they are written to the disk and persisted in memory. Upon opening the app, each view must be reregistered with the database. In the process of doing so, it will check for updates to the database and automatically pull any necessary changes.
Conclusion
Thanks for reading! I hope this gave you an understanding of Yap database and why it is such a powerful framework for iOS. I’ve just started writing technical posts, so please let me know if I can clarify or improve anything! For a more detailed explanation of any of the preceeding topics, check out the Yap docs.
