Firebase: Developing a Web Service with Admin SDK, Flask and Google Cloud
Firebase Admin SDKs enable accessing Firebase from trusted environments. Therefore Admin SDKs can be used to interact with Firebase from various public and private clouds. Incidentally, curated cloud platform offerings like Google Compute Engine, Google App Engine and Google Cloud Functions are excellent deployment targets for server-side Firebase apps based on Admin SDKs. These cloud environments provide developers with a wide range of options with respect to scale, convenience, DevOps automation and cost.
In this post we will develop a simple web service using Admin Python SDK and Flask. Like in most server-side development projects, we will first code and test the app locally. Then we will conduct a production drill by deploying the code in Google Compute Engine — Google’s infrastructure as as service (IaaS) cloud. We will learn how to access Firebase from a server-side Python app, and how to authorize Firebase API calls made by the code deployed in Google Cloud Platform (GCP). We will also explore some of the potential pitfalls, and come up with solutions.
Setting up the development environment
To start with we need a workstation with Python 2.7 or higher. I would also recommend using virtualenv to keep your project and its dependencies isolated from everything else. Execute the following commands in a Unix/Linux shell to start a new virtualenv sandbox, and install the required modules.
$ virtualenv env
$ source env/bin/activate
(env) $ pip install firebase-admin flaskWe use Google Application Default Credentials (ADC) to authorize the Firebase API calls made by our application. ADC is the recommended authorization mechanism for applications deployed in GCP. It saves us from having to hardcode credentials into the application, or ship a sensitive credentials file with the code. However, in order to locally test an application that uses ADC, we need to do a couple of things. First, download the service account credentials for your Firebase project. Then set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the downloaded file.
(env) $ export GOOGLE_APPLICATION_CREDENTIALS=path/to/creds.jsonFinally, we need to make sure that the Google Cloud SDK is installed in the system. This gives us the gcloud command-line tool. We will use it later to interact with Google Compute Engine.
Coding the app
Without further ado, lets go ahead and code our Python web service. If you are following along, you can simply copy the contents of listing 1 into a file named superheroes.py. Make sure to replace the <DB_NAME> placeholder at line 10, so it points to your Firebase database.
At line 9 of listing 1 we initialize the Firebase Admin SDK by calling initialize_app(). This is done outside of the functions, to ensure that it happens only once. Note that we are not passing any explicit credentials to initialize_app(). This prompts the SDK to look for ADC. Then we obtain a reference to a Realtime Database node called superheroes. All the data managed by our web service will be stored under this node. Next we have the four functions that constitute the public interface of our web service. Notice how they are mapped to different HTTP methods and URL paths using Flask decorators. Finally we have an internal helper function named _ensure_hero().
When deployed, the superheroes web service exposes four operations corresponding to the typical CRUD operations:
POST /heroes: Creates a new superhero entry.GET /heroes/<id>: Retrieves a superhero entry by ID.PUT /heroes/<id>: Updates a superhero entry by ID.DELETE /heroes/<id>: Deletes a superhero entry by ID.
Taking it for a spin
Now we are all set for a local trial run. So lets fire up that Flask!
(env) $ export FLASK_APP=superheroes.py
(env) $ flask runThis starts a web server that listens on port 5000. Lets try to create a new superhero in the database. Listing 2 shows the JSON payload that we will use. Simply copy it to a file named spiderman.json.
Now run the following curl command to POST the spiderman.json file to our web service. If all goes well it returns a 201 Created response containing a unique ID.
$ curl -v -X POST -d @spiderman.json -H "Content-type: application/json" http://localhost:5000/heroes
< HTTP/1.0 201 CREATED
< Content-Type: application/json
< Content-Length: 35
< Server: Werkzeug/0.13 Python/2.7.6
< Date: Fri, 15 Dec 2017 22:16:39 GMT
<
{
"id": "-L0RF2E2upW9jhCjmi6R"
}At this point you should also head back to the Firebase Console, and inspect the contents of your Realtime Database. A new entry is created under the superheroes/ path. You will notice that the key of the new entry is same as the ID returned by our web service. You can send this ID back to the web service to retrieve a superhero entry.
$ curl -v http://localhost:5000/heroes/-L0RF2E2upW9jhCjmi6R
< HTTP/1.0 200 OK
< Content-Type: application/json
< Content-Length: 197
< Server: Werkzeug/0.13 Python/2.7.6
< Date: Fri, 15 Dec 2017 22:18:11 GMT
<
{
"name": "Spider-Man",
"realName": "Peter Parker",
...Feel free to also experiment with PUT and DELETE requests. See how each operation mutates the contents in the Firebase database. If you send a request with an invalid superhero ID, the server responds back with a 404 Not Found response. This check is implemented in the _ensure_hero() helper function in listing 1.
$ curl -v http://localhost:5000/heroes/invalid_id
< HTTP/1.0 404 NOT FOUND
< Content-Type: text/html
< Content-Length: 233
< Server: Werkzeug/0.13 Python/2.7.6
< Date: Fri, 15 Dec 2017 22:18:16 GMT
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
...Deploying to Google Compute Engine: Take 1
There’s a lot to consider when deploying a production web service to an IaaS cloud. How many VM instances to use? What is the size of a VM instance? How to handle load balancing? What sort of monitoring to use? How do we scale with varying load conditions? The list of questions can be truly staggering. For this reason, it is often simpler and better to deploy web-facing apps in a platform as a service (PaaS) cloud like Google App Engine. They handle all that heavy lifting for us (and then some). But IaaS has its uses, and purpose of this exercise is to see how we can access Firebase from Compute Engine. Therefore we will keep things absolutely simple, and just deploy our code to a single VM instance. In a future post I will discuss porting this web service to Google App Engine.
First thing’s first. We need to ensure that the gcloud command-line tool is configured to interact with the “correct” GCP project — especially if you have multiple projects in use. Recall that our application uses ADC to authorize Firebase interactions. For this to work, we should start our VM in the same GCP project as our Firebase database. A Firebase project is actually a GCP project in disguise — meaning when you created your Firebase project, you actually created a GCP project. We need to make sure that our Firebase database, and the Compute Engine VM instance reside in the same project. Understandably, ADC in project X cannot authorize access to a Firebase database in project Y — at least not without additional configuration. To specify the project for gcloud, find out your project ID from Firebase Console, and run the following command.
$ gcloud config set project my_firebase_project_idNow lets start a fresh Linux VM instance, and SSH into it. I will call this instance flask-demo.
$ gcloud compute instances create flask-demo
$ gcloud compute instances start flask-demo
$ gcloud compute instances ssh flask-demoCompute Engine instances are fairly minimalist. So you will have to do some work to get all the necessary software installed. Run the following commands in the VM.
(vm) $ sudo apt-get install -y python-pip
(vm) $ pip install --user firebase-admin google-auth-oauthlib
(vm) $ sudo pip install flaskA couple of things to note here. The package google-auth-oauthlib is not really required for our app. It was installed simply to overcome a crypto module loading issue. This is being fixed as I write this post, so chances are you won’t need it. Secondly, we install Flask globally so that we can run the flask command-line tool from the shell. Note that we are not using virtualenv here, but feel free to if you wish.
Once the VM instance is fully set up, copy the web service implementation over to the VM’s file system. You can scp the source file, or simply copy-and-paste the file contents manually since superheroes.py is small. In a real world setting you will have version control systems and other tools to handle such tasks. Finally, fire up that Flask!
(vm) $ export FLASK_APP=superheroes.py
(vm) $ flask runWe haven’t configured the firewall of the VM yet. Therefore we cannot test this by sending requests remotely. But we should be able to run a curl command on the VM itself. When we do, however, we get this.
< HTTP/1.0 500 INTERNAL SERVER ERROR
< Content-Type: text/html
< Content-Length: 291
< Server: Werkzeug/0.13 Python/2.7.13
< Date: Fri, 15 Dec 2017 01:47:01 GMT
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
...The Flask server also logs something like the following.
[2017-12-15 01:47:01,051] ERROR in app: Exception on /heroes/spiderman [GET]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/hkj/flask-demo/demo.py", line 23, in read_hero
hero = superheroes.child(id).get()
File "/usr/local/lib/python2.7/dist-packages/firebase_admin/db.py", line 147, in get
return self._client.body('get', self._add_suffix())
File "/usr/local/lib/python2.7/dist-packages/firebase_admin/_http_client.py", line 93, in body
resp = self.request(method, url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/firebase_admin/db.py", line 769, in request
raise ApiCallError(self._extract_error_message(error), error)
ApiCallError: 401 Client Error: Unauthorized for url: https://my-db-name.firebaseio.com/superheroes/-L0RF2E2upW9jhCjmi6R.json
Reason: Unauthorized request.
127.0.0.1 - - [15/Dec/2017 01:47:01] "GET /heroes/-L0RF2E2upW9jhCjmi6R HTTP/1.1" 500 -Oops! What happened? It seems our web service received a 401 Unauthorized response from the Firebase database. In other words, the ADC on our VM instance failed to authorize the Firebase database call. But why?
To fully comprehend what’s going on, we need to have a rough idea of what OAuth2 scopes are.
Setting OAuth2 scopes on the VM
Scopes provide a way to restrict the level of access a user gains from an OAuth2 token. By scoping a token down to a few services, a service provider can ensure that the bearer of the token can only access those services. This is quite useful in an environment like GCP where there are dozens of services. Scopes constitute a layer of defense against accidental or malicious abuse of cloud services. They also facilitate authorizing clients in a more fine-grained manner. For instance you can have a user with access to only Cloud Storage, and a user with access to only Cloud Firestore in the same GCP project. This is achieved by including the appropriate subset of scopes in the tokens issued to the two users.
Various GCP services and the OAuth2 scopes required to access them are documented here. To access Firebase database, client applications must present an OAuth2 token with the following two scopes:
https//www.googleapis.com/auth/firebase.databasehttps://www.googleapis.com/auth/userinfo.email
However, ADC in Compute Engine does not add these scopes to the issued tokens. Consequently, Firebase database rejects the requests made by our web service in the cloud. This in turns results in the error we have seen.
Each VM instance in Compute Engine has a fixed set of scopes associated with it. The ADC on a VM instance will only include these scopes in the OAuth2 tokens. Therefore the solution to our problem is to configure the VM instance with the required scopes. We can check what scopes it currently has by running the following command.
$ gcloud compute instances describe flask-demo --format json
...
"serviceAccounts": [
{
"email": "your.gserviceaccount.com",
"scopes": [
"https://www.googleapis.com/auth/cloud-platform",
"https://www.googleapis.com/auth/userinfo.email"
]
}
],
...The serviceAccounts section in the output shows the OAuth2 scopes currently set in the VM. We can see that a required Firebase scope is missing. So we need to stop the VM instance, add the required scopes, and start it again.
$ gcloud compute instances stop flask-demo
$ export SCOPES=https://www.googleapis.com/auth/firebase.database,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform
$ gcloud compute instances set-service-account flask-demo \
--scopes $SCOPES
$ gcloud compute instances start flask-demoIt is also possible to create a VM with the necessary scopes by passing the --scopes flag to the instance creation command. That can help us avoid many of these additional steps.
Deploying to Google Compute Engine: Take 2
Now we are ready for another test run in the cloud. Start the Flask app as usual, and try sending some requests. Everything should work out fine this time around. ADC in the reconfigured VM should make the required scopes available in the OAuth2 tokens.
You may also be interested in sending requests to the superheroes web service from a remote machine. To enable this, we need to do two things.
- Bind the Flask web service to the public facing network interface of the VM. This can be achieved by passing a flag to the Flask start up command.
(vm) $ flask run --host=0.0.0.0- Expose the Flask HTTP port (5000) from the VM’s firewall. This is done via
gcloudtools in your local development environment.
$ gcloud compute firewall-rules create open-flask-rule --allow tcp:5000 --source-tags=flask-demo --source-ranges=0.0.0.0/0Now our superheroes web service in the cloud can be accessed from anywhere, using the public IP address of the VM instance.
Conclusion
We successfully developed and deployed a server-side Firebase app in Google Compute Engine. Ours was a simple web service based on Python and Flask, but it could have been anything. Possibilities include:
- A Python Scikit-Learn script that periodically trains a model over Firebase data
- A Java JAX-RS API built on top of Cloud Firestore
- A Node.js Express server app for sending FCM notifications in bulk
- A standalone Go binary that incrementally updates the privileges of users
The development and deployment process we went through in this example was pretty straightforward. There were only two things to be mindful of, and both were related to our use of ADC:
- Start the VM instances in the same GCP project as the Firebase database. This is what allows us to use ADC in the first place.
- Assign the required Firebase OAuth2 scopes to the VM instance. This is required so that ADC on the VM can authorize the Firebase interactions made by our code.
I will soon write a post about deploying this application to Google App Engine. You will see that due to the higher level of abstraction, the deployment process is even simpler there. In the meantime you can find all the source and support material related to this mini project on GitHub.
I hope this gave you an idea of the type of things that can be built using the Firebase Admin SDKs, and how to deploy such applications in Google Cloud. As always I’m open to comments and questions. I’m also eager to learn what you are planning to build with Firebase and Google Cloud.
