What we’ve learned from four years of using Firebase
Since the beginning of SketchDeck we’ve used Firebase and been enthusiastic advocates of the technology.
Now that it’s been four years and Cloud Firestore’s just been released (essentially, Firebase 2.0) it’s a good time to reflect on our experience.
What’s worked well
Overall we’ve really enjoyed working with Firebase — it’s an exceptionally flexible and useful layer for building applications.
We’ve built lots of things on top of it — realtime chat, a project management platform, a billing system and a design collaboration tool.
Firebase has performed well and coped with all those very different demands. One of the things I appreciate about Firebase is that it grows well with the product development cycle; early on when you’re hacking together ideas you can use the Firebase editor to mock up data and get a feel for how your application is working, then later on you can use the same datastore to serve up gigabytes worth of data per month.
Realtime is an incredible product design default
My favorite thing about Firebase is that it strongly encourages you to make everything in your software realtime. When your application uses realtime data it’s a magical experience, no longer do you tell people “oh refresh the page…” after you do something for them, instead it appears in front of them as soon as you did it — it’s a really pleasant UX.
Realtime is one of those all-or-nothing things: as soon as one piece of your data doesn’t broadcast updates, you cannot easily join that with the realtime data and it’s tempting to give up on being realtime all-together.
Thanks to Firebase’s easy to use APIs and broad applicability, you can be fully realtime from the start and keep that property as you build out. Various times we’ve build parts of our platform to not be realtime (usually due to laziness) and almost always come to regret it and then re-written those components.
Effortless scale
We’ve gone from having a handful of test projects to having hundreds of thousands of real, live design projects without ever dealing with database servers: pretty incredible! Firebase has been our easiest to handle piece of infrastructure (if only SQL servers were so simple!).
There are some occasional short outages which bring our entire business down. These hurt a lot, but thankfully have been very infrequent and short. It’d be nice to have multiple providers to choose from to try to mitigate this central dependency.
You can safely build critical data structures in Firebase
It took me a while to build up to trusting Firebase for mission critical, fragile data structures (e.g. our billing system), but we got there and its been working very well. We use Firebase transactions to implement locking and secure certain areas of the database to only allow our RPC methods to modify them.
Firebase’s shortcomings
Whilst our overall experience has been very positive, no technology is perfect. We ran into various challenges adopting it, which I outline here.
Lack of tools
We first adopted Firebase not long after its launch and often had to build tools and work-arounds for missing features. Now, five years after launch, Firebase is still a new-comer to the database world and lacks the ecosystem many other databases benefit from.
Tools: Dashboarding/business intelligence (BI)
There are not many BI tools that support Firebase. We have all of our core business information in Firebase, and it’s essential we can query it and chart it.
To deal with this shortcoming we copy all of our Firebase data into MySQL and then run our BI off of that. We do this in a semi-realtime fashion, every time data is updated in Firebase it’s also broadcasted into Kafka and then consumers take the data from Kafka into MySQL. This works really well, but is a big piece of infrastructure we’ve had to build.
Similarly, Firebase lacks aggregation tools — for example, you cannot request the sum of all project prices, or perform a complex GROUP BY statement. We need to do these sorts of things all the time, so MySQL is essential. The downside to this infrastructure is that you lose the realtime benefits of Firebase.
We’ve found there are common infrastructure operations we need to do that no tool exists for:
Tools: Map / Data migration
We commonly want to run a “map” operation across a large collection of objects in Firebase. We frequently do this when running migrations or when fixing the effects of a bug. As our database is now 6 gigabytes, naively loading up an entire collection into memory doesn’t work very well. Instead we’ve built a distributed system to do this.
Our migration system has two components: a master process loads a Firebase location specified by the user and gets all the keys (using the “shallow” flag in the REST API). It then queues a message for every key in Amazon Simple Queue System.
A cluster of worker processes (managed by an Elastic Beanstalk, scaled on the size of the input queue) then spins up and runs our specified handler for every object a message has been created for.
Tools: Backup restoring
The next difficulty is extracting information from our backups. Occasionally we want to revert part of our database back to that of an automatic backup. Since our backups are 6 gigabyte JSON files, they are unwieldy to download/upload/open in a text editor.
To solve this we’ve built a streaming JSON parser in Node.js that will extract our desired sub-tree and write it out to a file. This still manages to take 45 minutes to process, so more work needs done!
Security rules are too fragile and limited
Security rules might be one of our biggest struggles with Firebase. The rules have been designed to be fast for Firebase’s servers to execute at request time, but the downside of this is the rules are limited that it’s very hard to build an enterprise platform on top of them.
On a recent call with another founder I remarked I was on the edge of writing a meta-language that compiles down to Firebase security rules and he laughed, telling me he’d already done so.
What we really need is a way to implement role based security, for example if we could execute set intersections in the security rule then we could better express things like “if the user has one of the skills required to load this resource, let them read it”.
Finally, the security rule json file quickly grows to be an unweildy size and it’s rather terrifying to open up the rules editor and touch it lest you break the rules and bring down your entire application.
Databases should have an audit layer
A common situation when helping one of our users is something’s a bit off or broken and we ask “how did this happen?”. To solve this, I now create “audit” sub-objects for our objects.
Every time the system or a user makes a change to an object, an entry is pushed to the audit log of what they did, when and why.
Given Firebase handles authentication, it could automatically build an audit log for every object. This’d be really valuable and save application builders a ton of time. Being able to re-wind back through time on an object could save us a substantial amount of debugging time.
Shortcomings that Firestore addresses
Querying is challenging and Indexing is a lot of effort
Firebase is based around a denormalised data approach — you want to Write-as-you-want-to-Read and thus your application loads really fast. The downside of this is you end up indexing data often, in many ways, to allow you to load it later. To achieve this you need to write a lot of indexors, who listen for data changes then transform and write it out to the database.
It’s exciting that Firestore addresses this weakness through much more powerful queries.
Native support of collections and objects
It’s a really common pattern in Firebase to have collections of objects — however, Firebase does not support the Array type due to its poor distributed properties (e.g. if two people insert at the same time, what happens?). Some Firebase libraries let you load a location up as an object or a list, then offering specialized methods to manipulate each.
Firestore natively supports this concept, which is exciting as then hopefully the whole library and tool ecosystem can provide more intuitive interfaces.
Looking ahead
Thanks Firebase team for releasing such a well designed and solid product, we’ve really enjoyed working with it. Whilst it does have shortcomings, Firestore looks set to solve a bunch of them and we’re excited to try it!
I hope you enjoyed reading this — as we try new things and learn from them we’ll continue to post those experiences here on Medium. If working for one of the most innovative design services in the world interests you, check out our jobs page, we’re hiring.
