Simplified Firestore with Redis
Firestore is great, but I wanted to get rid of some boilerplate, make it faster, and add validation.
Features
simple-cached-firestore offers a number of key features:
- transparent, no-effort Redis caching to improve speed and limit costs
- model validation (optional, suggest using validated-base)
- simplified API to reduce boilerplate
- still have access to the underlying firestore client if you need custom functionality
Why build an API when using Firestore?
Obviously one of the biggest and most popular features of Firebase/Firestore is that it can be used entirely serverless. With the correct configuration, it can be securely accessed directly from the web or a native app without having to write your own API.
But that comes with a few big sacrifices that I wasn’t willing to make.
Validation
You can’t easily validate your data models without an API. There is a capability to write rules, but I really don’t want to spend hours writing complicated validation logic in this DSL:
Furthermore, in some cases, it’s just not possible. If you have any kind of complicated validation logic, or even something as simple as wanting to use constants from a library, you’re out of luck.
Sanitization
Additionally, the rules merely determine whether or not to allow a write to occur.
What if the properties you are checking are valid, but the user has messed with the Javascript and is saving extra arbitrary properties within the same object? Or much more likely, what if you accidentally attach properties you don’t mean to save? In either case, you only have limited control over what gets written to your DB.
Caching
Caching can serve both as a circuit breaker, and insurance against malice or bugs. This is why it’s unfortunate that caching also cannot be implemented in a serverless setup without a lot of complexity.
When implemented well, caching provides significant benefits in terms of cost-reduction and responsiveness.
With simple-cached-firestore wrapper, I regularly see cache hit rates of 95–98%. Amounting to a huge reduction in Firestore READ costs.
Usage
Moving on to the subject at hand, we’ll look at how I’ve addressed the shortcomings above with an API and simple-cached-firestore.
Each instance of simple-cached-firestore is responsible for all reads and writes to a specific collection, and it’s assumed that all elements of that collection can be represented by the same model.
To create an instance of simple-cached-firestore, we must first create the model that will exist in the collection.
Create a Model
At a minimum, the model has to fulfill the following interface:
The easiest way to do this is to just extend validated-base (the subject of the post on validated models) and use that.
Now that we have a model to work with, let’s create an instance of simple-cached-firestore.
Create simple-cached-firestore
As mentioned above, a single instance is responsible for reading and writing to a specific Firestore collection.
Reads are cached for the configured TTL, and writes update the cache. Because all read and write pass-through this layer, cache invalidation is not an issue. We have perfect knowledge of what is written, so the only real limit on the cache TTL is how big of a Redis instance you want to pay for.
You may not want to do all of these operations in one place like this, but this is the general idea.
The validated class we created above serves as both validation of anything that’s passed to it, and a way to translate the object to and from the DB (and the cache) into a class instance with known properties.
Basic CRUD Operations
You can see the breakdown of the basic operations here, but included the expected create, get, patch, update, and remove.
To give you an idea of how these CRUD operations are implemented, here is an example of how simple-cached-firestore implements the get operation. It’s actually more complicated than this, but this is just to show the major details.
The full implementation is here, and includes some extra work with timestamps to avoid race conditions contaminating the cache. But basically the process is:
- Check cache and return if cache exists
- Otherwise get snapshot and convert into a model instance
- Update cache before returning if a value is found
Pretty straight-forward, and you can imagine write operations working in a similar way.
Depending on the problem you’re solving, and if you’re careful about how you design all of the data models for your project, you can actually do a large portion of the regular tasks with just the basic CRUD operations.
This is great if you can manage it because it not only minimizes costs in normal operation, but thanks to the cache, means that you’ll almost never have to hit the Firestore itself.
Query Operations
At some point, some type of query operation is usually required in most projects, even if it’s just a list operation with a single filter. In Firestore this is done by chaining operations, often in a specific order. In order to abstract and simplify this, I created a simpler query abstraction that looks like this:
In use, the query objects look like this:
One important thing to note is that while queries are cached, due to the complexity of the query logic, accurate invalidation is hard. As a result, the cache for queries within a given collection is invalidated on every write to that collection. This makes it not very useful by default, so if you want effective caching of queries, that should be implemented on a case-by-case basis.
Custom Functionality
If the crud and query functionality don’t work for you in a specific case, you can always access the underlying Firestore client or cache instance with:
But keep in mind, that any modifications you make directly to objects in Firestore will not be captured by the cache unless you update it manually, and can result in inconsistencies if you don’t do it properly.
Next
From here I’ll next describe how the validated models and simple-cached-firestore can be integrated together in a dependency-injected Node microservice architecture.
