Cloud-based Data Persistence in Flutter

Share my State

Disclaimer: I am not affiliated with Google in any way other than being a general consumer of their free services (e.g. Gmail).

What?

In my previous article we augmented the Stateless Flutter starter app, by adding local persistent-storage to our state’s data via the SharedPreferences package.

Our Flutter starter app now has a single screen with a counter in the center of the screen. The counter begins at zero, and via a “+” button in the lower-right it increments, by one, at each tap. We also added a reset-to-zero button, for a little variety. Every time we re-start the app, the counter picks up where we left off last time…well it does if:

• we’re using the same device/simulator
• we don’t uninstall the app from that device/simulator

There’s a further list of caveats to consider listed at the end of the previous article that might be too limiting for your longer term app aspirations.

In this episode we’ll replace SharedPreferences by instead storing our state data in a Cloud-based database — specifically Google Cloud Firestore. We’ll demonstrate how this also gives us the ability to share data across multiple devices concurrently.

Here’s a peek at what our app will look like when we’re finished this article. To highlight the new shared-data capability, we demonstrate with both an iOS simulator and an Android Emulator running the same code, side-by-side. Each tap on the increment (or reset) button of one device is quickly reflected on both devices:

The GitHub repo for the finished product above is at https://github.com/eggzotic/flutter_persistent_cloud. But don’t go there yet. As stated in the earlier episodes, the learning is in the journey we follow below. Besides that, if you take my repo as-is, it is missing the Google Firestore plist/JSON files — you need to create your own Firestore Project and DB so that you will have access to the console for the backend-setup, and the ability to add your own iOS and Android apps. So treat the repo as a reference for debugging your source code only.

Why?

Making data available beyond the lifetime of an app invocation, as we did in the last article, is just step one.

Storing small amounts of data directly on-device is handy for quick access and re-use — but it may not be scalable (or appreciated by the users!) as the volume of data grows. Trusting data-storage to each user’s device can be a risk. If an app’s data includes larger objects such as images or video then storing on-device may not be ideal. If the user’s device is lost or destroyed then their data will be gone with it.

Making data outlive even the installation of an app on a user-device may be desirable (or required). The aggregate data (i.e. the combined data from all your app users) of your app may need to be analysed, or stored centrally for some compliance requirement — this cannot be done if the data is individually stored on each user-device only.

Many apps enable a user to create data, and then make that data available for other users of the app to consume, e.g. can anyone say “Social Media”?

For this functionality we need to store data where it can easily be made available to all app-users, subject to suitable access-related logic.

How?

In this article, for no particular reason (other than the author’s familiarity with it) above other equivalent options, we’ll use “Google Cloud Firestore” to store our state data off-device. Firestore is a sophisticated product, but one I’ve found very useful for real world apps. Setup is more involved than on-device options, but the payback is that we also get to use more sophisticated features for enhanced functionality in our apps. Firestore is also free for small-to-medium scale projects.

No longer will we entrust the durability of our state data to the device on which it is created, and where its availability beyond that device is effectively nil.

But really, how?

As we did in the previous article we will use the finished product from there as our starting point. In case you don’t have it already, you can pull it from the repo. Rename the top-level folder to our new project name: flutter_persistent_cloud.

Clutter Cleanup

Open the pubspec.yaml file and update the name and description lines at the top of the file. Once you have updated the name (e.g. to flutter_persistent_cloud), you may also need to update the test/widget_test.dart file where an import line will be broken due to the package name change. Then run this to sync up:

flutter packages get

Due to our re-using the finished result of previous projects as the starting-point for this project, there is some embedded config within our iOS and Android folders that is now a bit off. So we will take the sledgehammer approach and simply delete those folders entirely, and subsequently use flutter to recreate them. This demonstrates a handy feature of flutter that can be used if either of the build or platform folders (ios, android), ever become corrupted when you’re working on an app.

Simply remove those folders. e.g. On Linux or Mac OS you can do it like so:

cd flutter_persistent_cloud
rm -rf ios android build

Then re-create these with “fresh” versions:

cd ..
flutter create flutter_persistent_cloud
cd flutter_persistent_cloud

Note that in this situation, flutter create … will re-create the missing folders (ios, android, build) only rather than creating a completely new default project. It will not touch the existing contents of the lib/ folder (all our source code!) or pubspec.yaml file.

Create the Firestore Project

In your browser, open the Firebase Console and login to your Google account (if not done automatically). Here you should be greeted with “Your Firebase projects”. Click the link to create/add a new project. Follow the process of:

1. Enter your project name: e.g. FlutterPersistentCloud
2. Disable Google Analytics for this project
3. Create Project
4. Wait while a new Firebase Project is provisioned.
5. Select “Continue” to return to your “Project Overview”
6. Various features to add are presented: select “Cloud Firestore”.
7. Under the “Cloud Firestore”, select “Create database”.
8. Accept the default, “Start in Production mode”
9. Select a “Cloud Firestore location” near you (or just accept the default)

From the main database screen select “Start collection” to create an initial collection named “counterApp”:

For the first document, give it the name “shared”:

• Name the first field “counterValue”, of type “number” with value 0
• Also add a field “deviceName”, of type “string”, with value “No device”

We now have our basic persistent data storage in the cloud all setup and initialised.

Set the Database Access Rules

For a fully fledged multi-user app Google Cloud Firestore provides full user-authentication management features. However, user-authentication is beyond the scope of this article. So to satisfy the requirements for our demo-app (and to safeguard our new Firestore database from possible abuse), we will simply limit access to the single document “shared”. There is a size-limit of 1 MB, and a limit of 20000 fields, per document with Cloud Firestore.

While we’re still in the “Database” section of our Project on the Firebase Console, select the “Rules” tab. Since we chose the (default) production mode, all access to the database is initially denied, as can be seen from the starting access-ruleset:

rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /{document=**} {
      allow read, write: if false;
    }
  }
}

Update the rules, by adding the italicised lines below, just above that original match statement:

rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /counterApp/shared {
      allow read, write: if true;
    }
    match /{document=**} {
      allow read, write: if false;
    }
  }
}

Then select “Publish” to activate the new rules.

Be sure that the path (i.e. /counterApp/shared) in your new match statement above matches exactly the collection name and document names you created earlier — the path matching is case-sensitive.

These new rules allow anyone access to just that single document “shared” that we created earlier.

Add a Mobile App to Firebase Project

Time to add an Android and/or iOS app to the Firebase Project. For a production Flutter app it is likely you will want to add both mobile platforms. However, for a learning exercise like this, you might choose to focus on just the one most convenient for you. We’ll cover both cases here — pick whichever you prefer (or both!).

The primary documentation guides for Firebase are targeted at native iOS and Android apps (along with Web, and others). Flutter instructions are included in another part of the site. Below is a distilled set of steps that I have found to be reliable. We will use the Cloud Firestore package.

The instructions that follow are based on the “Setup”, for each mobile platform, at the Cloud Firestore package site along with a couple of tweaks that I have found to smooth the path for each mobile platform.

To add an iOS App

If you intend to run your app on Android-only — then skip this section.

In your Firebase Console, select “Project Overview”, then select the iOS icon above “Add an app to get started”.

We need to fill-in the iOS Bundle ID of our project. This can be extracted purely from the command-line, or from Xcode.

Method 1: At the Mac terminal, from the project top-level folder, copy and paste this command (as a single line):

grep PRODUCT_BUNDLE_IDENTIFIER ios/Runner.xcodeproj/project.pbxproj | head -1 | awk '{ print $3 }' | cut -f1 -d ';'

In my case the output to the above is: software.eggzotic.flutterPersistentCloud

Then copy and paste your output into the “iOS bundle ID” field at the “Register app” step in your browser.

Method 2: use the Xcode GUI to find the bundle ID. At the Mac terminal, from the project top-level folder run this to open Xcode in the correct folder:

open ios/Runner.xcworkspace

Select “Runner” in the top-left of the Project Navigator. In the “General” pane, at the “Identity” section copy the value in the “Bundle Identifier” field. Then paste this value into the “iOS bundle ID” field at the “Register app” step in your browser.

Regardless of which method you used above, continue here: Ignore the optional “App nickname” and “App Store ID” fields. Click “Register app”.

Click the button/link “Download GoogleService-Info.plist”. This file will now appear in your browser Downloads folder. Locate the file and, if necessary, rename it to exactly “GoogleService-Info.plist”. E.g. if you already had a file by that name present, the new one may have an added “ (1)” in the name — which needs to be corrected.

If Xcode is not already opened, open it as above for finding the Bundle ID. Drag and drop this GoogleService-Info.plist into the Runner folder of Xcode’s Project Navigator, right above Main.storyboard. Accept the defaults of the Xcode confirmation dialog. We can quit Xcode at this point — never needing to return!

Note that this is where we stop following the steps on the Firebase Console site. We do not follow the “Add Firebase SDK” and subsequent steps that would be done for a pure native iOS App. Flutter takes over from here. Just select the big “X” in the top-left of the web-page to return to the “Project Overview” screen.

To add an Android App

If you intend to run your app on iOS-only — then skip this section.

From your “Project Overview”, select the Android icon above “Add an app to get started” — or select “Add app” and then the Android icon if you’ve already added an iOS app.

We need to fill-in the Android package name of our project. This can be extracted purely from the command-line, or from Android Studio. Personally, I find the command-line method more straight forward here — having never become very familiar with Android Studio. I’ll leave the Android Studio method as an exercise for the reader — although it is most likely the more practical option for Windows Users.

On Linux or Mac copy-n-paste the below command into the terminal (as a single line):

grep -i applicationID android/app/build.gradle | awk '{ print $2 }' | cut -f2 -d\"

In my case, the output to the above is:

software.eggzotic.flutter_persistent_cloud

Then copy and paste your output into the “Android package name” field at the “Register app” step in your browser.

Ignore the optional “App nickname” and “Debug signing certificate SHA-1” fields. Click “Register app”.

Click the button/link “Download google-services.json”. This file will now appear in your browser Downloads folder. Locate the file and, if necessary, rename it to exactly “google-services.json”. E.g. if you already had a file by that name present, the new one may have an added “ (1)” in the name — which needs to be corrected.

Move the file from your Downloads folder into the android/app folder of your flutter project.

Edit the file android/build.gradle — find the buildscript section and check that the repositories subsection contains the line:

google()

Also check that the allprojects section, repositories subsection, includes the line:

google()

In the buildscript section, add this line to the dependencies subsection:

classpath 'com.google.gms:google-services:4.3.3'  // Google Services

Edit the file android/app/build.gradle. Find the line:

apply plugin: 'com.android.application'

and below it add this single line:

apply plugin: 'com.google.gms.google-services'  // Google Play services

Note that this is where we stop following the steps on the Firebase Console site. We do not follow the “Add Firebase SDK” and subsequent steps that would be done for a pure native Android App. Flutter takes over from here. Just select the big “X” in the top-left of the web-page to return to the “Project Overview” screen.

Extra required step for Android — we need to update the file android/app/build.gradle and change this line:

minSdkVersion 16

to instead be:

minSdkVersion 21

for compatibility with the Google Cloud Firestore package during build. This effectively excludes some older versions of Android, but this should not be a real problem for modern devices.

All Platforms

Now that we have made the platform-specific mods to accommodate our mobile-platform(s) of choice, we can begin work on the Flutter project.

Add these lines to our pubspec.yaml file in the dependencies section:

cloud_firestore: ^0.12.11
device_info: ^0.4.1+3

and remove/comment-out the line referring to shared_preferences. Then run

flutter packages get

To the code changes! We will introduce a new class to enable extracting our device name. Create the file lib/my_device.dart and add this content:

import 'dart:io';
import 'package:device_info/device_info.dart';
import 'package:flutter/material.dart';
// helper class to get my device name
class MyDevice with ChangeNotifier {
  // Constructor
  MyDevice() {
    _setup();
  }
  //
  // instance properties
  String _name;
  String get name => _name;
  //
  // transient properties
  bool _isWaiting = true;
  bool get isWaiting => _isWaiting;
  //
  void _setup() async {
    final devInfo = DeviceInfoPlugin();
    if (Platform.isAndroid) {
      final androidInfo = await devInfo.androidInfo;
      _name = androidInfo.host;
    } else if (Platform.isIOS) {
      final iosInfo = await devInfo.iosInfo;
      _name = iosInfo.name;
    } else {
      _name = 'Other';
    }
    _isWaiting = false;
    notifyListeners();
  }
}

Notice that second line import — we’re using the DeviceInfo package which provides the friendly device-name. We’ll use that information to record the device which last updated the counter. Since our new project will be sharing the data with all other devices running the app, it will give each user a little insight as to who made the most recent update. This is a compromise between not giving away too much personally-identifiable data, while still helping to highlight how we’re sharing data across devices.

Note that in this case we’re deliberately using not-the-latest-version of DeviceInfo, so as to be compatible with Flutter 1.9. If you’re using Flutter 1.10+ already then you should be good to use the very latest version.

The import for dart.io is required to give us access to the Platform class, which helps us identify which platform we’re on at runtime.

Finally, notice that we use ChangeNotifier with our new class MyDevice since it involves async operations and we need to notify our listeners when the device name has been set. Again we use a transient state property _isWaiting (with an accompanying public getter) to indicate whether the instance is ready.

Coming back to lib/counter_state.dart, remove the import for shared_preferences. Add these imports, in its place:

import 'package:cloud_firestore/cloud_firestore.dart';
import 'my_device.dart';

Remove the shared-prefs key static property we had before:

static const _sharedPrefsKey = 'counterState';

and replace it with these lines which will provide us the location in the database of the data-document as well as the field names under which our data is stored:

// the location of the document containing our state data
final _sharedCounterDoc = Firestore.instance.collection('counterApp').document('shared');
// convenience, to avoid using these string-literals more than once
static const _dbCounterValueField = 'counterValue';
static const _dbDeviceNameField = 'deviceName';

Again, note that the string literals above must match exactly the collection and document names we created in Firestore and set in the DB access rules. Any mismatches among those three (DB structure, DB access-rules and code) will result in the app failing to function correctly).

Find the declaration of _value:

int _value;

and add these lines below it, to store the info relating to the device which last updated the counter, as well as our own device name:

String _lastUpdatedByDevice;
MyDevice _myDevice;

Below the getter for value, add these new getters so we can access the name of the last device to update the counter as well as our own device name from the UI:

String get lastUpdatedByDevice => _lastUpdatedByDevice;
String get myDevice => _myDevice.name;

Also, find the definition of the getter isWaiting:

bool get isWaiting => _isWaiting;

and update it to take into account whether _myDevice is ready:

bool get isWaiting => _isWaiting || _myDevice.isWaiting;

We will completely separate the operations of “save” and “load”. We can delete the current definitions of _save() and _load():

void _load() => _store(load: true);
void _save() => _store();

Our previous _store() will be used for save only. So rename it to _save().

We’ll update the _save() method like so:

• The optional load boolean can be removed from the parameter list — we’ll no longer use that.
• We can remove the artificial delays (i.e. Future.delayed) because we will be interacting with a real network call out to Google Cloud Firestore.
• The content of the try clause will become a single call to DocumentReference.setData with a Map literal containing both the counter value (_value) and our device-name (_myDeviceName)

So our _save() method will be as below. If you put it side-by-side with the original _store() that we started out with, hopefully the similarity will be clear.

//  save the updated _value to the DB
void _save() async {
  _hasError = false;
  _isWaiting = true;
  notifyListeners();
  try {
    await _sharedCounterDoc.setData({
      _dbCounterValueField: _value,
      _dbDeviceNameField: myDevice,
    });
    _hasError = false;
  } catch (error) {
    _hasError = true;
  }
  _isWaiting = false;
  notifyListeners();
}

For loading/receiving data we will use a feature of Firestore that enables us to receive updates whenever there are changes in the content (or even existence) of a document. This method is DocumentReference.snapshots() and it returns a Stream of DocumentSnapshot objects. Since we have a stream, we can use the standard method of adding a Stream listener to define actions to be performed every time updates are received. Each time the document is updated then all currently listening devices will receive new data in the form of a DocumentSnapshot object which encapsulates the current state of the document and some metadata such as the document ID, whether the data came from local-cache (more on that later), and whether the data even exists!

We’ll call this method _listenForUpdates() and here it is:

// how we receive data from the DB, and notify
void _listenForUpdates() {
  // listen to the stream of updates (e.g. due to other devices)
  _sharedCounterDoc.snapshots().listen(
    (snapshot) {
      _isWaiting = false;
      if (!snapshot.exists) {
        _hasError = true;
        notifyListeners();
        return;
      }
      _hasError = false;
      // Don't trust what we receive
      // - convert to string, then try to extract a number
      final counterText = (snapshot.data[_dbCounterValueField] ?? 0).toString();
      // last resort is use 0
      _value = int.tryParse(counterText) ?? 0;
      _lastUpdatedByDevice = (snapshot.data[_dbDeviceNameField] ?? 'No device').toString();
      notifyListeners();
    },
    onError: (error) {
      _hasError = true;
      notifyListeners();
    },
  );
}

• The block of code inside .listen(…) is executed every time an update is received from Firestore — regardless of who changed the data. The updatedDocumentSnapshot object (called snapshot, here) is passed to the block and processed, with all conclusions resulting in notifyListeners() being called to notify the UI.
• In the above you can see we begin the actions by declaring that we’re no longer waiting — all of these actions are synchronous after all.
• We check that there is some data (snapshot.exists is true) otherwise we notify that an error has occurred and return from this update.
• we then extract the new values for _value and _lastUpdatedByDevice from the Map that is contained in snapshot.data.
• note that there is currently a quirk (or bug?) with the CloudFirestore packages where DocumentReference.snapshots() call will silently receive nothing (and never receive updates, as if there was just no data to be found) if permission is denied (due to access rules on the DB) . So if you can see in the Firebase Console that data is there but your code is receiving no data — this may be the cause. In that case your app will likely be stuck at “Please wait…”. Using DocumentReference.get() would throw an Exception (e.g. as a once-off initial load action) that can be communicated to the user. Review the DB access rules in this case. This subtle issue is being tracked on GitHub here and here.

Finally, we need to update the CounterState constructor. Previously we had a lone call to _load(). Instead, we now need to initialise _myDevice and then begin listening for data from the DB. Recall that the constructor for the MyDevice class itself contains an async private method _setup() which calls notifyListeners() when the required values are accessible. So, in our CounterState constructor we must add listeners to _myDevice so that we can notify the UI that the data is ready to display. Here’s how we do that:

CounterState() {
  _myDevice = MyDevice()
    ..addListener(() {
      notifyListeners();
    });
  // start listening for DB data
  _listenForUpdates();
}

The ..addListener(…, in this case, is a short-hand for _myDevice.addListener(… (and the previous line would have to be terminated with a semi-colon). Writing it as above highlights that the listener needs to be added immediately so as to not miss any notifications that _myDevice may emit.

We’ll make some updates to lib/main.dart to better reflect the new shared setting that our app is in. Update the MyApp title line:

title: 'Local Storage Demo',

to instead be:

title: 'Cloud Storage Demo',

Similarly update the MyHomePage instantiation line:

child: MyHomePage(title: 'Local Storage Demo'),

to instead be:

child: MyHomePage(title: 'Cloud Storage Demo'),

Alter the text indicating the meaning of the counter:

: counterState.isWaiting ? 'Please wait...' : 'You have pushed the button this many times:',

to be like so:

: counterState.isWaiting ? 'Please wait...' : 'The counter value is:',

Finally, we’ll add two new lines of text indicating:

• which device last update the counter, and
• the name of the device we’re running on right now

Find these lines in the children of the main Column:

counterState.hasError
  ? Text("Oops, something's wrong!")
  : counterState.isWaiting
    ? CircularProgressIndicator()
    : Text(
      '${counterState.value}',
      style: Theme.of(context).textTheme.display1,
    ),

and add these lines directly below. Note that we’re adding a Column inside the already existing Column. This is an example of the unlimited ability to compose widgets from other widgets:

(counterState.hasError || counterState.isWaiting)
  ? Text('')
  : Column(
    children: [
      Text('last changed by: ${counterState.lastUpdatedByDevice}'),
      SizedBox(height: 16.0),
      Text('(This device: ${counterState.myDevice})'),
    ],
  ),

So we are ready to run the code. Go ahead and start an Android Emulator or iOS Simulator, and do our first run:

flutter run

That first run may take a few minutes to complete the build, and bring up the app on the sim screen, as there is a lot of first-time compilation of the Firestore package. All going well you can now click the “+” button and see our counter increment. (If you’re trying to build on iOS and hitting an error related to deprecated timestampsInSnapshotsEnabled, then check the solution here relating to pod update ….)

Time to share that data! Quit the flutter run and start up a second simulator/emulator concurrently — either of the same platform or one of each. Depending on your desktop/laptop memory and horsepower, you may even start three simulators. Ensure that flutter can see all of them in the output of:

flutter devices

Then run on all of them:

flutter run -d all

Then from any one of them click the increment button and notice how all devices update with the shared-counter value, within a moment. They also all indicate which device made the most recent update. Additionally, if you have your Firestore Database “Data” tab open in the browser, with the “shared” document selected you will see the data being updated when any device makes an update.

Note that, from the Firestore console in the browser, we can delete the individual database document fields, or even the entire document, the app will be notified immediately and react gracefully. Subsequently recreating the document will also cause the running app to react and become functional again.

The finished source code is available in my GitHub repo. Remember that really only the source code (lib/ folder) and pubspec.yaml are of any use as you need to create your own Firestore project, database and access-rules as we walked thru above.

Fun Firestore Facts

So we have stepped up the persistent state storage — moving the authoritative source of our data into the Cloud. But…you may notice that since we removed the artificial-delay code, that you barely see the CircularProgressIndicator in the UI any more (i.e. we don’t seem to be waiting, as one might expect for such long-distance communications). There are a few reasons for this:

• The data-payload for this app (i.e. a number and a device-name-string) is really very small — nowhere near the 1 MB limit per document — so we’re not taxing the network in that sense.
• Our code has separated the operations of storing a new value (via _save()) and receiving updates (via _listenForUpdates()).
• Since processing updates is synchronous from the moment they arrive, only _save() actually involves any waiting-for-completion, and so it is the only one that may trigger the waiting-indicator.
• Firestore by default, and completely transparently to our code, caches data locally on the device, meaning that “writes” (i.e. calls to setData()) complete very quickly without waiting for the data to traverse the network to the Firestore servers, or the corresponding acknowledgement. So it will be taking little more time/effort (for the local-write) than it did when we used SharedPreferences (and in that case we also saw no effective waiting time without including the artificial delay). The code considers the write complete once it is committed to local cache — hence the quick save.
• Despite the local-cache providing the illusion that the data update is very fast, the reality for other devices is that they’ll not see the new data before the remote-write to the Firestore servers has completed. Hence, as you can see in the demo GIF above, one device can update the number multiple times before another device may see those updates.

Our code is not transactionally safe. Firestore does support transactions that ensure only a single device is updating a database document at any point in time — forcing others to wait while the transaction completes. However for simplicity we have not covered that here and so we’re operating on a “last in wins” basis — the most recent update will be the one propagated. This may not be what you want in some cases! Be aware and use transactions if you need them.

Another Firestore feature we’ve not touched on in this article is offline-support. Again, by default and completely transparently, when your device cannot reach the Firestore servers, it will use (for both read and write) data from the local-cache, and automatically flush the writes to Cloud Firestore when the network connection is available again (unless it has never cached this data before (for reads), in which case you may receive errors). This may also cause some writes to arrive at the Firestore servers very late and overwrite an earlier (higher!) value. Note that transactions do not function when offline and will always cause your code to throw an error — so be ready to catch.

If you intend to store your new project in a GitHub repo then it’s best to exclude the Firestore files. They do not contain super-sensitive data but will trigger warnings from some of the security features of GitHub that scan for API keys in public repos. To do that, assuming you have already run git init, then add these lines to .gitignore:

# Google Firebase / Firestore related
**/ios/Runner/GoogleService-Info.plist
**/android/app/google-services.json

If you had already added those files (e.g. if you had run git add --all), then you can remove them from the repo while leaving them on-disk:

git rm --cached ios/Runner/GoogleService-Info.plist
git rm --cached android/app/google-services.json

Subsequent git commit and git push commands will ensure that those files are not present on your public GitHub repo.

As mentioned earlier, for a scalable app with even modest data capacity and complexity requirements, database-access will most likely require the use of user-authentication, which can then be referenced in your Database “Rules”. Creating and managing users and passwords, and authentication methods with Firestore is available through the Firebase Console web portal. However the code to manage authenticating the user in the app is more complex and worth a separate article….now there’s an idea for a future article — stay tuned!

Previous episodes: Local Data Persistence in Flutter, Stateless Flutter