Firestore and Cloud functions in Golang — the unknown issues

Frikky
Frikky
Nov 20, 2019 · 12 min read

With the usability of Firestore and Cloud functions getting better and better, I have finally had time to dabble with ways they can be used together. After all, these products are amazing, but have some caveats that I will try to shed some light on, especially for Golang. I still use both of these features in a production environment, but had to do some workarounds to get there. My personal reason for writing this piece is specifically because of issues with rollbacks and infinite loops.

If you have no previous experience with how GCP, Firebase and Golang interactions, I suggest reading the following articles first, but it’s not a necessity (you might not understand what’s happening otherwise, as I skip a lot of essentials).

Integration possibilities

  1. Traditional: User → Backend → Database (firestore)
Traditional view of how to work with a backend on serverless infrastructure.

2. New approach: User → Firestore → Backend → Firestore

View of user talking directly to the database, triggering a cloud function to run further updates.

We will focus on the latter approach, as the former uses a traditional backend & database setup.

Getting started

  1. Trigger possibilities for our cloud function
  2. Firestore’s storage format(s)
  3. Reverting and rolling back data

If you want to follow along, I assume you have a project ready already in Firebase, otherwise:

  1. Create a new one: https://console.firebase.google.com/u/1/
  2. Enable the “firestore” database
  3. Enable billing (required for cloud functions, but FREE)

If you want to jump straight into things, here is some ready code:

1. Firestore cloud function triggers

gcloud config set project YOUR-PROJECT

We’ll run an example where a user changes their username, and we want to verify whether the rest of the data (e.g. their email) is valid (the same). We will be updating an existing Document /users/{userid}, before revering it if the information is wrong. Let’s get to some actual code.

The code below is the baseline we’ll start with. Take note that the “main” function is called HandleUserChange. This is the function itself that we want to deploy, and is also an argument to gcloud CLI when deploying a function.

package user

import (
"context"
"errors"
"log"
"time"

"cloud.google.com/go/firestore"
)

var client *firestore.Client

type FirestoreEvent struct {
OldValue FirestoreValue `json:"oldValue"`
Value FirestoreValue `json:"value"`
UpdateMask struct {
FieldPaths []string `json:"fieldPaths"`
} `json:"updateMask"`
}

type FirestoreValue struct {
CreateTime time.Time `json:"createTime"`
Name string `json:"name"`
UpdateTime time.Time `json:"updateTime"`
Fields User `json:"fields"`
}

// This is our self-defined fields.
// FirestoreEvent.Value.Fields = User
type User struct {
Username StringValue `json:"userId"`
Email StringValue `json:"email"`
DateEdited IntegerValue `json:"date_edited"`
}

type IntegerValue struct {
IntegerValue string `json:"integerValue"`
}

type StringValue struct {
StringValue string `json:"stringValue"`
}

// Simple init to have a firestore client available
func init() {
ctx := context.Background()
var err error
// FIXME - add your username in here
client, err = firestore.NewClient(ctx, "medium-77273")
if err != nil {
log.Fatalf("Firestore: %v", err)
}
}

// Handles the rollback to a previous document
func handleRollback(ctx context.Context, e FirestoreEvent) error {
return errors.New("Should have rolled back to a previous version")
}

// The function that runs with the cloud function itself
func HandleUserChange(ctx context.Context, e FirestoreEvent) error {
// This is the data that's in the database itself
newFields := e.Value.Fields
oldFields := e.OldValue.Fields

// As our goal is simply to check if the username has changed
if newFields.Username.StringValue == oldFields.Username.StringValue {
log.Printf("Bad username: %s - %s", newFields.Username.StringValue, oldFields.Username.StringValue)
return handleRollback(ctx, e)
}

// Check if the email is the same as previously
if newFields.Email.StringValue != oldFields.Email.StringValue {
log.Printf("Bad email: %s - %s", newFields.Email.StringValue, oldFields.Email.StringValue)
return handleRollback(ctx, e)
}

return nil
}

To deploy it, set up the file above (full version here), before running the command line below. Take not that we only look for “document.update” in the trigger-event and not “document.create” or similar. Make sure to change “YOUR-PROJECT” in both the

gcloud functions deploy HandleUserChange --runtime go111 \
--trigger-event providers/cloud.firestore/eventTypes/document.update \
--trigger-resource "projects/{YOUR-PROJECT}/databases/(default)/documents/users/{userid}"

When the process is done, make sure it exists in firebase by going to the following link, where “YOUR-PROJECT” is your project ID: https://console.firebase.google.com/u/1/project/YOUR-PROJECT/functions/list. It should look a little something like this:

A Golang cloud function in the Firebase view

Now, lets make some example data that we can change to trigger the function. Go to https://console.firebase.google.com/u/1/project/YOUR-PROJECT/database/firestore/data (change YOUR-PROJECT) and set up the following:

Creation of a new collection and document in Firestore

With data available, we can now change a value in the database, and the function should trigger. To see whether it works or not, try to change the “email” field and you should see the following in your logs.

Example view of logs when the email failed (as in the code described earlier). I failed to set my billing account :)

With a function execution tested, lets move on to the real issue(s).

2. Firestore’s storage value format

# Data FROM firestore on trigger
# Expected
{"fields": {"userId": "username"}}
# What we really get
{"fields": {"userId": {"stringValue": "username"}}}
# Data expected when SENDING TO firestore (updates)
# Expected because of the above
{"fields": {"userId": {"stringValue": "new_username"}}}
# What firestore expects
{"fields": {"userId": "new_username"}}

What this means for you is that you always have to keep two formats in mind. I’m not sure what the design choice was behind this, or whether I’m just doing it wrong, but I got it working after a while. I’m sure there are available reflection tricks and struct tag control, but I haven’t found any good way. Anyway, let’s move to some examples.

Example 1 — simple structure: User struct vs Firestore User struct

The following structure is a basic way of using Firestore triggers in Go. This might not look too bad, but when you move into more complex structures such as maps and arrays, this gets out of hand. This gets especially annoying when you want to send the data back to Firestore, as it doesn’t expect this format.

// Default, what I would expect you have from other places
type User struct {
Username string `json:"userId"`
Email string `json:"email"`
DateEdited int64 `json:"date_edited"`
}
user := User{Username: “username”}
// What firestore gives you (everything below)
type User struct {
Username StringValue `json:"userId"`
Email StringValue `json:"email"`
DateEdited IntegerValue `json:"date_edited"`
}
type IntegerValue struct {
IntegerValue string `json:"integerValue"`
}
type StringValue struct {
StringValue string `json:"stringValue"`
}
user := User{Username: StringValue{StringValue: “username”}}

Example 2— complex structure: User struct with subfield Access array subfield

Now look at this structure. Both parts are identical (top and bottom), but the overhead on the latter part (because of firestore) is tremendous. Not to mention; how would you possibly translate the second structure back to the first, original one? I really hope I am missing something, and this wasn’t their design choice.

// Again, this is the structure we want to represent
type User struct {
Username string `json:"userId"`
Email string `json:"email"`
DateEdited int64 `json:"date_edited"`
Access []Access `json:"access"`
}
type Access struct {
Id string `json:"id"`
}
user := User{
Username: "username",
Access: []Access{
Id: "id",
},
}
// Now look how out of hand this got all of a sudden. To represent the information above,
type User struct {
Username StringValue `json:"userId"`
Email StringValue `json:"email"`
DateEdited IntegerValue `json:"date_edited"`
AccessValue AccessValue `json:"access"`
}
type AccessWrapper struct {
ArrayValue ArrayValue `json:"arrayValue"`
}
type Access struct {
Id StringValue `json:"id"`
}
type ArrayValue struct {
Values []Value `json:"values"`
}
type Value struct {
MapValue MapValue `json:"mapValue,omitempty"`
StringValue StringValue `json:"stringValue,omitempty"`
IntegerValue IntegerValue `json:"integerValue,omitempty"`
ArrayValue ArrayValue `json:"arrayValue,omitempty"`
}
type IntegerValue struct {
IntegerValue string `json:"integerValue"`
}
type StringValue struct {
StringValue string `json:"stringValue"`
}
type MapValue struct {
Fields interface{} `json:"fields"`
}
user := User{
Username: StringValue{
StringValue: "username",
},
AccessWrapper: AccessWrapper{
ArrayValue: ArrayValue{
Values: []Value{
Value{
MapValue: MapValue{
Field: Access{
Id: StringValue{
StringValue: "id",
},
},
},
},
},
},
}
}

With that quick introduction to how the datatypes interact, the next section will build upon our existing user.go file, and add functionality for rollbacks as that’s the reason for this blogpost itself.

3. Reverting / rolling back to oldValue

  1. A user updates location document X (e.g. their own user)
  2. You check whether the data is valid or not in a cloud function
  3. If the data is valid, you update document Y (some other location) or do nothing, then return nil. If the data is invalid however, you have to roll back (either update or delete → create).

Now imagine having to rollback, and the structure above having to be decreased from the latter (complex) to the first (simple), and having 20 database collections to manage. That’s not user friendly to a developer at all (horrible UX to be honest). You would have to have both structs available, and make a copy just to roll it back. One of the reasons for this design might be how they use transactions, which allows you to update a single field really easily, but that doesn’t help us in this situation.

To work around having to do all this manual work, I instead wrote some (horrible) code using reflection and recursion, which I made into a library over here: https://github.com/frikky/firestore-rollback-go

Here is what it does:

  • Reduces the complex structure from data unreadable to Firestore to normal (what you would expect)
  • Gives you a single call to roll back data or get the required interface.
  • Doesn’t require definition of any other than your own data
  • Doesn’t require you to know what structure to translate to
  • Doesn’t require double struct definitions

Caveats:

  • Doesn’t translate normal JSON struct to a Firestore struct yet
  • Having to redefine your struct from e.g. IntegerValue to rollback.IntegerValue

How you would normally do it for each structure (Remember: you might have a lot of big complex structs):

// Handles the rollback to a previous document
func handleRollbackUser(ctx context.Context, e FirestoreEvent) error {
// This can be grabbed from split of e.OldValue.Name
clientDoc := client.Collection("users").Doc(strings.Split(e.OldValue.Name, "/")[6])

// Again, you have to
type NewUser struct {
Username string `json:"userId"`
Email string `json:"email"`
DateEdited int64 `json:"date_edited"`
}

// Might crash on this
newDate, err := strconv.Atoi(e.OldValue.Fields.DateEdited.IntegerValue)
if err != nil {
return err
}

translatedUser := NewUser{
Username: e.OldValue.Fields.Username.StringValue,
Email: e.OldValue.Fields.Email.StringValue,
DateEdited: int64(newDate),
}

setter, err := clientDoc.Set(ctx, translatedUser)
if err != nil {
return err
}

log.Printf("Successfully updated document. Data: %#v", setter)
return errors.New("Should have rolled back to a previous version")
}

Instead, what you can do with the library, having a single generic rollback.

import "github.com/frikky/firestore-rollback-go"

// This is our self-defined fields.
// FirestoreEvent.Value.Fields = User
type User struct {
Username rollback.StringValue `json:"userId"`
Email rollback.StringValue `json:"email"`
DateEdited rollback.IntegerValue `json:"date_edited"`
}

// Handles the rollback of any previous document
func handleRollback(ctx context.Context, e FirestoreEvent) error {
// writtenData, firestoreReturn, err :=
_, _, err := rollback.Rollback(
ctx,
client, // Your firestore client
e.OldValue.Name, // The path to roll back
e.OldValue.Fields, // The parsed data you have
)

return err
}

You can now write rollback.StringValue instead of defining them yourself, as well as have a single function handles your rollback for you. No dealing with type casting etc. If you would just like the data without a rollback, run this instead:

parsed, err := rollback.GetInterface(yourStructValue)

Infinite loops

Another workaround for this might be to delete the document, then run a new creation which you don’t have a listener for, but my current implementations check whether the new data is newer than the old data with the DateEdited field. I’m not sure which approach is best, but I suspect the latter if you don’t have issues with write amounts to Firestore itself.

Try it yourself

To test it locally without a cloud function, use the file “user.go” and edit the values in main (yes, I’m aware this is horrible testing). If you want a ready file to deploy to Firebase, use the folder “functions”, containing “user.go” again, as well as a deploy script “deploy_firebase.sh” (edit it with your function). The file test.go contains some more information that might be interesting. Tree:

.                          # Tree
├── function # A function that's ready
│ ├── deploy_firebase.sh # Helps you deploy
│ ├── go.mod
│ ├── go.sum
│ └── user.go # A ready-made go file for deployment
├── test.go # Test file that might be useful?
└── user.go # A local testing field

Remember to change every instance of “YOUR-PROJECT” before attempting to deploy. When you run it, you can look at the database directly in the Firebase view, as your connection to it is also part of the realtime feature, meaning you can see the data updating in real-time.

Summary

Good about all of this: I learned more about reflection and recursion.

Follow me on Twitter for more things like this, and check out my other blog posts on Golang, serverless and random things @Frikky

Happy serverless coding :)

Update: This blog was written for the website Niceable, which is a work in progress where I test out anything serverless.

The Startup

Get smarter at building your thing. Join The Startup’s +800K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store