Firebase Data Within A Team
Why Reside Uses Fireadmin
The Problem
At Reside, a tech-enabled real estate brokerage, we deal with large amounts of mission critical data associated with home sales, which we store in Firebase’s Real-Time Database (RTDB) and Firestore. As engineers within our team work on adding features to our existing production systems, data often needs to be migrated between Firebase projects.
The migrations include:
- Copying data from Production to another project for testing
- Changing the structure of existing data within Production
- On-boarding new engineers as they join the team
- Loading a data backup into a project
Initially, we would create simple scripts locally (for simplicity we called it “the data tool”) which used the Firebase Admin SDK. These scripts would choose which project to update based on details in a service account JSON file sitting in the same directory. In order to point the script to the right Firebase project, we would simply switch out whichever copy of the file was located in the working directory. This worked alright, but made it easy to make mistakes such as pointing to the wrong Firebase project.
For a while we lived with “be careful when using the data tool” mentality. That lead to yours truly accidentally deleting some production data (fun way to break a sweat). Thankfully, we run backups often and could quickly recover.
Even though things were able to be solved quickly, this situation conflicted with one of Reside Engineering’s core values:
Our team members should have the space to build anything possible without the fear of impacting our production systems.
We needed a way perform actions, above board, making it easy for anyone on the engineering team to understand/track.
The Solution
After experiencing a similar pain with other Firebase projects in the past, I began crafting an application called Fireadmin. It also happens to work perfectly for our flow at Reside. It provides:
- Trackable system actions (data migration, bucket config changes)
- Actions that can be re-run on different Firebase projects
- Management of service accounts, leaving engineers free to copy data into their environments for fast development
Note
This article will cover why and how Reside started using Fireadmin. For a guide on how to setup Fireadmin, check out this article.
Environments
What is an environment?
In order to offer a stable application while still quickly adding features it is common to split the project into multiple “environments”. An environment is a version of your application at a certain stage of development such as Production, Staging, or Development.
With an application that uses Firebase, one of the most common ways of managing environments is by having a separate Firebase project for each environment. That way the resources for each environment, including RTDB, Authentication, Storage and Firestore, are completely separated from one another.
Development Flow
Our environments are matched directly to git branches. As an engineer completes a feature, they make a pull request against master (our integration environment). After another engineer reviews the code by running it on their dev environment, the pull request is merged into master.
Our QA engineers verify all new features on our integration environment throughout our sprint, confirming that they work together as expected and do not cause regression. This environment sees tons of testing traffic, which means data does not often directly match what is production.
As the sprint wraps up, we move into staging, which directly mirrors data on production. This allows us to confirm features work as expected with migrated legacy data.
What does that mean for data flow?
After an engineer completes a feature on their development environment, the following steps are taken:
- Engineer provides settings to be run in a Fireadmin action or just note “re-run the Copy Collection I just ran”
- After the pull request is verified, the engineer merges the code and runs the same Fireadmin action with integration environment as the destination
- The engineer notes within the feature ticket that an action run is required (we also add it to something we call “weekly deployment notes”)
- When sprint is wrapped up, actions mentioned in closed tickets can be run to prime the staging environment (which is again verified by QA before production deployment)
More Integration!
Our production deploy is on a weekly cadence, but sometimes our larger features need longer than that to develop. However, we don’t want to slow down our normal integration work-flow for smaller features, so we needed to come up with another solution.
We addressed this by setting up multiple integration environments (naming them int-b and int-c) . Each one is merged back into master as the feature set is completed:
Thanks to Fireadmin, developers can prepare their own environments to test features as if they were deployed to any one of the other environments. Once the branch has been merged back into master, we run an action to copy stage data back into the environment associated with master. That action is then re-run on both int-b and int-c.
On-boarding New Engineers
When a new engineer joins our team, part of the on-boarding process includes the engineer running a “setup migration” action. The action primes the database (both RTDB and Firestore) of the new Firebase project (the engineer’s “dev environment”) with data from our staging environment.
Using this pattern, the new engineer can immediately begin exploring the application’s features since they have a complete dataset. They can also safely begin building features with real data right away without breaking anything.
In The Future
So far, we have covered how Reside makes use of Fireadmin to make database migrations (both RTDB and Firestore), but it can also be used for actions between other cloud resources such as Google Cloud Storage. We are also exited about features planned for the near future of Fireadmin including User/Team Role Management as well as Protected Environments (environments that can only be modified with approval from a manager).
Conclusion
We hope you found this useful. Our team strives to help give back to the community, which has given us all of the many great tools we use.
Thanks to Lizzie Paris, Jeff Judkins, and Peter Miles for reviewing drafts of this post. Also — thanks to all of my teammates for the help testing and integrating Fireadmin into our development flow.
Also, if you haven’t heard of Reside, you should check us out. We are a growing team with a great culture… and we are hiring.
