Firebase: Developing an App Engine service with Python and Cloud Firestore

In my previous post I showed how to deploy a Python web service using Google Cloud Functions. That service used the Firebase Admin SDK to read and write to Firebase Realtime Database. In this post we are going to migrate the same service over to the Google App Engine (GAE) standard environment. But more importantly, we are going to up the ante by using Google Cloud Firestore instead of the Realtime Database.

This is a big deal because developers could not access Cloud Firestore from the GAE standard environment for a long time. Python client for Google Cloud Firestore uses multithreading, which is not allowed in GAE standard. So what changed?

Google Cloud Platform recently introduced the second generation Python 3.7 runtime for GAE. This new runtime doesn’t have the threading restrictions of the old Python 2.7 runtime. It also relaxes several other pesky constraints that used to frequently drive developers up walls. Specifically, the new runtime allows developers to use any dependency, directly access remote endpoints, and even access the /tmp directory of the local filesystem. The restrictions in the old runtime, which dates back to around 2008, were there for good reasons — mostly to ensure the security and isolation of the apps running on a shared infrastructure. But since then containers and user-space kernel technologies like gVisor have come into play. As a result, developers can now enjoy a lot more freedom and power, while retaining the same levels of security and isolation as before.

With that in mind, lets go ahead and implement a web service that connects to Firestore from GAE standard environment.

Setting up the development environment

You will need a billing enabled Firebase project, and a local installation of the gcloud command-line tool. If your project is brand new, make sure to enable Firestore support and App Engine support for it as well. You will also require Python 3 and the virtualenv utility. Execute the following commands in a Linux/Unix shell to get the development environment set up.

$ gcloud config set project <your-project-id>
$ virtualenv -p python3 env
$ source env/bin/activate
(env) $ mkdir heroes
(env) $ cd heroes/

These commands create a new virtualenv sandbox for your app, and create a new empty directory named heroes. We will use this directory to house all the source files associated with your service.

Next, you need to set up Google Application Default Credentials. This enables you to locally test the service before deploying it to GAE. Local testing does create some extra work, but in the long run it saves a lot of time and effort. Download a service account JSON file for your Firebase project, and set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to it.

(env) $ export GOOGLE_APPLICATION_CREDENTIALS=path/to/creds.json

The service account JSON file should not be in the same directory as your application. Keep it separately to make sure that you don’t accidentally upload it to the cloud.

Implementing the service

Our web service will consist of just 3 files:

  • requirements.txt — declares the required dependencies
  • main.py — contains the Python source code that implements our service
  • app.yaml — a GAE deployment descriptor

Lets start by creating the requirements.txt file. In the past developers had to vendor in all the dependencies into a separate subdirectory, and upload it as part of the app. But with the new runtime we just declare the dependencies in a requirements.txt file. GAE automatically fetches and installs the dependencies declared in this manner. Listing 1 shows what this file should look like for our example.

Listing 1: Dependency declaration (requirements.txt)

Normally you don’t have to declare google-cloud-firestore as a dependency. The Firebase Admin SDK should automatically install that for you. But there’s a known issue caused by a bug in pip, which sometimes prevents this from working correctly. The easiest way to work around this glitch is to declare google-cloud-firestore as a direct dependency. Once you have listing 1 in place, run the following command to install the dependencies in your development environment so that you can test the service locally.

(env) $ pip install -r requirements.txt

Now it’s time to code our web service. We use Flask to handle all the request routing, and the Firebase Admin SDK to initialize a Firestore client instance. listing 2 shows the complete implementation.

Listing 2: Web service implementation (main.py)

Note that this is nearly identical to the Python app we implemented for the Google Compute Engine environment several months ago. In fact, if you replace the Realtime Database calls in that implementation with Firestore calls, you end up with listing 2. The __main__ portion only gets called during local testing. In the production environment, GAE will provision a proper application server like gunicorn for you.

Finally, we create the GAE deployment descriptor named app.yaml. The only important thing to note here is the runtime: python37 entry. Listing 3 shows the final result.

Listing 3: GAE deployment descriptor (app.yaml)

Testing it locally

Time to take it for a spin. Simply fire off main.py as follows to get the service up and running.

(env) $ python main.py

This starts a web server that listens on port 8080. You can send it a few requests to make sure that everything is working as expected.

(env) $ curl -v -X POST -d '{"name":"Spider-Man"}' -H "Content-type: application/json" http://localhost:8080/heroes
< HTTP/1.0 201 CREATED
< Content-Type: application/json
< Content-Length: 35
< Server: Werkzeug/0.14.1 Python/3.6.1
< Date: Fri, 07 Sep 2018 22:54:57 GMT
<
{
"id": "oit8FKTbPVHgrtTtgKZ7"
}
(env) $ curl -v http://localhost:8080/heroes/oit8FKTbPVHgrtTtgKZ7
< HTTP/1.0 200 OK
< Content-Type: application/json
< Content-Length: 35
< Server: Werkzeug/0.14.1 Python/3.6.1
< Date: Fri, 07 Sep 2018 22:55:30 GMT
<
{
"name": "Spider-Man"
}

As you interact with the web service, check the contents of your Cloud Firestore database via the Firebase console. You should see the data in the superheroes collection changing in response to your requests.

Deploying to GAE

Use the gcloud command-line utility to deploy the code to GAE. Run the following command from the same directory as your app.yaml file.

(env) $ gcloud app deploy

The deployment takes a couple of minutes. Recall that GAE has to install the dependencies in the cloud based on the requirements.txt file. When finished, the CLI logs the URL of the live app.

Beginning deployment of service [heroes]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 2 files to Google Cloud Storage ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [heroes]...done.
Setting traffic split for service [heroes]...done.
Deployed service [heroes] to [https://heroes-dot-my-project.appspot.com]

Lets close things off by making a few HTTP calls to make sure that everything is in working order. The first request is likely to take an extra second or two, since GAE lazy loads your application on first request.

(env) $ curl -v -X POST -d '{"name":"Spider-Man"}' -H "Content-type: application/json" https://heroes-dot-my-project.appspot.com/heroes
< HTTP/2 201
< content-type: application/json
< date: Fri, 07 Sep 2018 23:00:23 GMT
< server: Google Frontend
...
<
{"id":"ZWJcJNCLSw7L2WxOqguQ"}

You can run the following command to stream application logs directly on your console:

(env) $ gcloud app logs tail -s heroes
Waiting for new log entries...
2018-09-07 23:00:18 heroes[20180907t155731] "POST /heroes HTTP/1.1" 201
2018-09-07 23:01:41 heroes[20180907t155731] "GET /heroes/ZWJcJNCLSw7L2WxOqguQ HTTP/1.1" 200
2018-09-07 23:01:53 heroes[20180907t155731] "DELETE /heroes/ZWJcJNCLSw7L2WxOqguQ HTTP/1.1" 200

Conclusion

I’ve been a long time user of Google App Engine. Actually, I studied its architecture and operation for my graduate research (e.g.). In many ways GAE is the technology that brought PaaS and serverless into mainstream. It’s refreshing to see how the different GAE language runtimes have evolved over time. They have relaxed many of their past restrictions, thus giving developers more freedom. At the same time, GAE as a platform has significantly improved its support for application governance via better tooling, monitoring, logging, and modularization.

The new Python 3.7 runtime in GAE is a great platform for building microservices that interact with Firebase. I hope you find it as powerful and exciting as I did.