The challenges of making a true Offline First app
In our recent hackaton we pushed hard with the Immer team to rebuild Idea Growr to add online accounts. We had some challenges regarding ‘Offline First’, but found a new path.
This article is part of a series of articles related to growing my idea growing app.
Updated October 2020: added RxDB reference.
Update June 2023: I’m working on a mobile wireframing app . The article continues below the video showing TinyUX.
Taking it online, while keeping it offline
In earlier articles I’ve written about the the conceptual aspects of the app. Now we can get into the actual building of an improved app.
From a technical perspective, the current offline android app (from now on I’ll call Classic) is quite simple. You store texts, images and audio and are able to retrieve and edit them.
The requirements for the new app (later I’ll call it Cloud) app are:
- Keep classic features, this includes being able to use the app without an account and while being offline.
- Be able to use an account and synchronize between devices.
- Use the same source code for iOS (iPhone, iPad) as for Android
Being able to use the app offline is often called an ‘offline first’ approach to mobile app development. We’ll get back to that.
The Hackaton
The plan in short; Build the core of the cloud version of the app during a hackaton week (5 days) with the full Immer team. I work at Immer and recently we decided to partner on the development.
In order to re-use the same code to build for iOS and Android, we decided to build the app in React Native. We use Expo as a layer on top React Native to avoid some complexity. We’ll see how far that will take us.
Some of us have some experience with Expo, others don’t. So besides rebuilding Idea Growr, the week is also a great learning experience and should be fun.
In app architecture
We wanted to keep all the logic & queries at a single container level, to make it easy to test and reason about the app.
In order to decide on the best architecture, I decided we needed a Proof of Concept (POC). I’ll explain the inner workings of the POC, but first a quick introduction of our team.
Team Immer
In order to refresh my mind from programming, I like to draw. After the hackaton this resulted in a bunch of illustrations. So lets put those illustrations to use and introduce you to the team.
Bas
Co-founder of Immer. Quick thinker, who came back to the Netherlands for the hackaton and then returned to his grand tour of Europe in his camper with his family. You can see where he does his remote work on this map.
Michael
Co-founder of Immer. Drupal expert, football enthusiast. Also attending the User Experience (UX) design and business side of the app.
Vincent
Co-founder of Immer. Responsible for marketing and sales, and also has decades of experience programming. During the week he and Maurits turned out to be our personal trainer on the beach.
Maurits
Dev-ops. Responsible for the architecture and tooling. Made the one-text-app POC in preparation. Can stand on a single leg, bend knee to fully drop down and move back up.
Marc
Like me, a developer with a User Experience design (UX) background. His dog frequents the Immer office. He’s experienced in GraphQL and Expo, so he also functioned as a support for others.
Akko
Former cook, normally responsible for service support & drupal development. He wasn’t able to join for the full week, but helped me out with localization and took charge of BBQ.
Ivo
Our developer with most React experience, loves to go low level. Correct indenting is of profound importance to his spirit.
Maarten
The friendliest developer. In a lighthearted grumpy fashion.
Erik-Jan
First week at the job (!), back-end / dev-ops guy. Great week to get to know him.
Me, Julius
Yours truly. I have a background in UX design and came up with Idea Growr back in 2013.
Proof of concept
In preparation of the week I wanted a proof of concept (POC) app to potential problems early and start the week with a repository of code that is structured correctly and ready for all to hit the ground running.
1-text-app
The POC was meant to be as simple as possible, but touch on the most important challenges.
The requirements:
- Edit a single text.
- Can edit offline and without an account, but also be able to create an account and login.
- When logged in, you should able to sync the text between devices.
Maurits was a bit time constrained in the weeks leading up to the hackaton. He was able create a POC that allowed for accounts, online storage and syncing of a text.
The POC was able to sync every X seconds and Maurits did some research on GraphQL caching and we decided to use this for local persistent storage. We then have no need for Redux (Often used for complex state management), simplifying our setup.
The setup
Maurits made a really nice setup that allowed everyone to get developing reasonably quickly.
He decided on using Litsor as a layer between the database, because it allows for the use of GraphQL. GraphQL is a simple to user query protocol, and the integration with React Native via Apollo include some caching capabilities that seem like a good fit for Offline requirements.
The Expo server runs on your laptop and because Litsor and the mySQL database run in Docker, the developer is not depending on an internet connection and can test everything locally.
Basic screens & components
In preparation I also made most of the important screens. Pressing a button wouldn’t do anything, but it would remove the mundane work from the hackaton in order to focus on the real challenges.
Expectations & hope
My predictions, written beforehand.
Best case scenario:
In one week everything is done for iOS beta, we even did some research on subscriptions and played around with creating a slack bot.
Also we found a workflow that gives us the confidence that we can produce mobile apps using these technologies and be ruthless efficient in how fast we can build and keep the quality high.
Worst case scenario:
The team gets frustrated and seeks refuge by hanging out at the beach and nothing of importance gets done. Those who do work, get irritated. The things they try to do fail, because Expo provides limitations we can’t really work around. So wee need to eject, then the remaining progress slows down because every code update takes minutes to compile. In the end, no app, no happy people, but some lessons learned.
As you may expect, the truth is often somewhere in the middle.
The week
This is what happened.
We all went to work, taking our Gitlab issue list I prepared as inspiration and splitting up in teams of 1 or 2 . Then we just went at it.
I had the role as product owner, so a big chunck of my time was spent moving around answering questions on how the app was supposed to work.
We where able to make great progress on a number of screens; creating an idea, creating a text note, localization, audio, images, lists of ideas with sorting, etc.
We discussed the limitations of Expo, but kept using it. I’ve had a short Twitter conversation with Brent Vatne that goes to the heart of it. In short; it seems that for payments we need to ‘eject’ eventually.
Lessons Learned & and revised future plans
While I see myself as a positive human being, I do like to focus on what goes wrong and should be fixed. Preferably in a structural manner. The biggest challenge now seems to be the first requirement:
Keep classic features, this includes being able to use the app without an account and while being offline.
Because we have over 6.000 existing users on Android we want to keep, this is a very hard requirement. In hindsight it seems we tried to to do too much at once. We should have developed Offline Only, before we tried Offline First.
Offline First, or Second?
A new way of thinking is starting to emerge in the web and mobile development scene, called Offline First.
We live in a disconnected & battery powered world, but our technology and best practices are a leftover from the always connected & steadily powered past.
Offline capability is a key characteristic of modern Progressive Web Applications. Offline first thinking must learn from and further what we’ve seen work with Responsive and Mobile First thinking.
On closer inspection, they really are talking about what I call Offline Second. A web-app dealing with connectivity dropping, should at first work online.
My definition of Offline First, means you first build it offline. Without internet, without online accounts. Then when the whole app works, build on top of this code to add online capabilities second.
There is an architectural challenge in creating an offline first app; how do you synchronize the data? I’ll describe two different strategies in as much as a non-technical way as possible.
Strategy 1: Caching queries & using resolvers
One way to look at is, is to see the online database as the sole truth. You can ask questions (queries) to this database, and when the connection drops, you locally remember what the answers to those queries used to be. You use the cache storage to keep answers to the database around.
We use Apollo for our data management and have to be careful about how to use this cache. When you are offline, the last answer to a question is not always the correct answer.
To deal with this problem, you have to do some coding by hand for each query. You write resolvers that make sure the correct thing happens. The conversation then becomes like this:
After this conversation, the server could come back online and the local storage of answers to the queries would be used to sync the data to the server.
The nasty thing about this approach seems to be that you have to do create resolvers for every query and slowly start to build a local database in the form of resolvers. A local database full of entanglements that is hard to manage.
My biggest hope 🙏 is that I’m ignorant and someone comments that we just need to do X and Y to get a clean manageable architecture.
Strategy 2: Local database & communicate the difference to the server
While the first strategy leans on providing the online server with a list of changes to the local understanding of the database, the second strategy is to use an actual local database.
Of of the advantages is that it simplifies offline behavior. The app just uses a routine way to talk with the local database (SQL queries). Then a general piece of software, I’ll call a syncer, is placed between the local database and the online server. It keeps track of what is out of sync and makes updates if required. It wouldn’t matter what SQL queries you write to make changes to the local database, so now resolvers, the syncer would just look at the state of the database to see what is different.
This kind of solution reminds me a lot like how how version control like Git works for code management. Git is a tool that keps track of your code and make sure you have a copy on your computer that you can use offline, but is also able to sync with the code on a server.
You to start coding locally and then tack on Git aftwerwards. In the same way I’d like to first create an app using the local database, and then tack on a syncer later.
Some potential candidates
While I have a preference for the second strategy, we have to do more research. Perhaps there are more elegant solutions.
Some candidates:
GUN is a small, easy, and fast data sync and storage system that runs everywhere JavaScript does. The aim of GUN is to let you focus on the data that needs to stored, loaded, and shared in your app without worrying about servers, network calls, databases, or tracking offline changes or concurrency conflicts.
I only learned about its existence during the hackaton. Skimming the texts, it’s hard to wrap my head around the impact of it being peer-to-peer.
GUN is fully decentralized (peer-to-peer or multi-master), meaning that changes are not controlled by a centralized server.
Firebase gives you functionality like analytics, databases, messaging and crash reporting so you can move quickly and focus on your users.
With offline persistence enabled, the Cloud Firestore client library automatically manages online and offline data access and synchronizes local data when the device is back online.
That ‘back online’ makes me feel it’s Offline Second. This quora page suggests it’s not really meant for Offline Only.
GraphQL resolvers with Apollo
Perhaps we missed something on how to make this work reliably, and we should just continue on the path we have taken.
This also looks promising, because the way they write about it, it actually is Offline First the way I defined it.
The Couch Replication Protocol lets your data flow seamlessly between server clusters to mobile phones and web browsers, enabling a compelling offline-first user-experience while maintaining high performance and strong reliability. CouchDB comes with a developer-friendly query language, and optionally MapReduce for simple, efficient, and comprehensive data retrieval.
It seems designed for data clusters, but then also mentions offline-first, so I’m somewhat confused on how it actually works. This article goes into detail how it could work with React Native, so i’ll have to look into that more.
Added October 2020: RxDB:
https://github.com/pubkey/rxdb
A realtime Database for JavaScript Applications
RxDB (short for Reactive Database) is a NoSQL-database for JavaScript Applications like Websites, hybrid Apps, Electron-Apps, Progressive Web Apps and NodeJs. Reactive means that you can not only query the current state, but subscribe to all state changes like the result of a query or even a single field of a document. This is great for UI-based realtime applications in way that makes it easy to develop and also has great performance benefits. To replicate data between your clients and server, RxDB provides modules for realtime replication with any CouchDB compliant endpoint and also with custom GraphQL endpoints.
Desired process
What could we have done better? It seems we may have tried to tackle to much at once. The ideal process keeps the momentum. At every step our team enjoys visible progress and we can keep learning and improving.
In hindsight, I should have aimed for the team to build an Offline Only app. We did a short evaluation, and this is our new roadmap:
- Offline Only (simple Sqlite database)
Get the React Native app as soon as possible in the Apple store so we can start learning and improving. This means we start storing the ideas on the device only. While the classic Android app is also Offline Only, we don’t have an iOs app yet, so we can roll it out there first. - Back-up the full database
The current would be greatly helped if they could have a simple back-up feature for all their ideas. Now there is a quite tedious export/import feature. A decent back-up feature would give us the opportunity to learn about our users needs in this area and dealing with the server side, analytics, etc. - Sync
Once we have the core working, we can build smart sync capabilities on top of that. Even before that, we can start experimenting with this right now in our one-text-app POC. - Teamwork
Then we move forward along the path discussed in earlier articles. Starting with teamwork.
There you have it
We had fun, made good progress, took two step forward and one step back. We calibrated our plans and now focus hard on first taking the app to the Apple Store as soon as possible.
If you have any suggestions for us on the Offline First challenge, please share your wisdom in the comments. I’ll keep this article up to date, incorporating new insights from you and others.
Update: The iOS version is live! ..and it’s no longer live, but you can read how it went.
Update November 2022 — Wireframing on your mobile phone.
Let me also quickly show you my new app for mobile wireframing.
Learn more at https://tinyux.app
Love to get your feedback!
