Python App Engine 2017: Building a simple Flask app
This article is part of a series where I’m intending to dig into App Engine deeply, investigating what’s really going on under the covers by testing hypotheses, performing experiments.
To do that I need a simple App Engine application that I can add experiments to, then run them and get results.
Also, in this previous article I wrote about getting up to speed with the recent changes in App Engine. I’ve been using App Engine for years, but I haven’t paid attention to changes in tooling and techniques until now, so I’m making an effort to get up to date.
If I’m going to build an hypothesis testing app, why not use the latest recommended App Engine techniques, and maybe even learn something along the way?
Having worked through the Quickstart guide, I saw this:
urh, what’s a Flask?
I’ve been using webapp2 until now. I’ve never heard of this Flask thing. Maybe the next bit of the docs, Getting Started with Flask on App Engine Standard Environment is illuminating?
Nope, I still don’t know what the taters are. Ok Google, what’s the taters?
Googling gives me this link about Flask. Apparently it’s a whole thing…
Oh, a microframework based Werkzeug, Jinja 2 and good intentions? uh. Well, I do know what Jinja 2 is, that’s the templating system I’ve been using with webapp2. Nice.
What does Flask’s webpage say?
Oh, Flask is fun! Well, you should have told me that before!
But that little code snippet tells me a lot.
Apparently I can just make a Flask “app” object (hopefully a WSGI app?) by passing in my module name (because why?). But what’s cool is that you can decorate a function with a route and turn it into a web handler:
@app.route("/")
def hello():
return "Hello World!"Really? That’s pretty awesome. In webapp2, you have to make a stodgy class, descending from webapp2.requesthandler, and it all feels a bit more clunky than this. I actually don’t like using classes all that much in Python; they’re a necessity in App Engine particularly for ndb, but I’d prefer to stay in pure function land. I’ll have more to say about that in subsequent articles. But for now, nice.
Ok, so how do we do this in App Engine? Let’s go work through Getting Started with Flask on App Engine Standard Environment.
Getting Started with Flask on App Engine Standard Environment
Ok. In the Quickstart article I managed to get 1 and 2 done. Let’s look at 3.
First of all, Eclipse and PyDev? I know a lot of people will grumble about those, but that’s what I already use. Tops. However, since I installed gcloud I don’t have things quite hooked up correctly; the servicesdemo project has a lot of red underlines and none of the nice IDE features work because I haven’t got source paths and etc set up.
Ok. So I’m going through Getting Started with Flask on App Engine Standard Environment. Yup, got python, got eclipse, got pydev. Now to create a project:
Cool, ok, let’s create one of those. This is going to be my hypothesis testing app, and it’s going to be deployed to my emlyn-experiments project. So let’s just call it emlyn-experiments, ok.
Now the interesting bit:
Ok Google, where on God’s green earth are my App Engine source folders?
So before gcloud, I downloaded the App Engine sdk, put it in a folder, and you know, that’s where it was. So this step was easy. But gcloud installed as a package in that a-la-peanut-butter-sandwiches magic debian package way. So, where’d it go? Let’s click “Where is it?” in the docs:
Ok Google, that’s bullshit.
A bit of googling, I can’t find anyone that knows. Stack Overflow doesn’t even seem to know.
Ok, let’s just fall back to scratching around in the filesystem like a rodent, dammit.
I found something promising here: /usr/lib/google-cloud-sdk/platform/google_appengine
You can see me pasting in the folder, and BAM, we have liftoff. Let’s just assume this is correct and I’ll go crying later if it’s wrong.
Ok, the next bit says we’re going to make a user comment form, but I don’t really want to. Let’s follow the guide, but create the hypothesis testing app.
The Hypothesis Testing App
Before I keep going, I’d better have a little thinky about what this app will be.
It needs to do let me add experiments, run them, and get results.
There are a lot of ways this could be. An heuristic I use is, when you have multiple options and you don’t know which to use, do the simplest thing.
The simplest thing would be this:
There is a main page, which is just a list of links, a big switchboard. Each is an experiment.
To add an experiment, I add code to the app and deploy it. That should include adding a link for that experiment somehow.
To run the experiment, I click the link.
How to see results? I think I can assume there’ll be a GCP Datastore object which is some kind of top level reporting object for an experiment. A generic way to view this object would be to have it dump out a JSON representation of itself, and slap that on a page. I could just override to_dict().
So I’ll need a report page, which takes a datastore object key, retrieves the object, calls to_dict(), and shows me the result. If the experiment is underway, I can refresh the page to get an update.
Then, the link that kicks off an experiment should:
- kick off the experiment
- send me to the report page with the key for an object that’ll show me results.
So, for this Flask app, I’ll need two horrible server side pages. That sounds doable!
Building the app
So let’s look at this Flask quickstart guide, see how to do this.
Ok. The Hypothesis Testing app can use this structure, but I’ll need to modify it a little.
I probably wont use any CSS (because Times New Roman is how I roll).
I’ll need two templates, one for the experiment switchboard and one for the report page.
I might break main.py out into multiple files (switchboard and report).
And I’ll be needing a background service separate from the default service, so I’ll need background.yaml, and a queue.yaml to define the background queue (see my post on Services for a quick explanation of this, lol).
I’ll create a bunch of blank files for now. Here:
What’s the next bit? Oh, “Setting up libraries to enable development”. Ok. And a quick read…
Vendoring!
So, Python has a whole package management system, as do many languages. It’s normal to install packages using pip install. They go somewhere magic on your machine, and Python can just find them.
However, App Engine doesn’t support that. Instead, we need to add packages to our code base, and deploy them along with the rest of our code.
The package management system supports installing packages to a specific directory instead of to the magic place. Then, in your App Engine code, you need to do a thing called vendoring to make Python find these packages at run time.
When I first started with App Engine, I didn’t know about vendoring, and couldn’t make head or tail of how to make Python’s import mechanisms do anything smart. Python’s import mechanisms drive me crazy generally, they have turned my hair grey and made most of it fall out.
Then, I found out about vendoring, and have slowly been converting over to it. But it was a custom solution by a third party developer, involving arcane python files and holding your mouth just so as you sprinkle the material components into the sacrificial urn.
Now, here’s Google just casually telling me that I’m Doing It Wrong:
So apparently there’s a built in library, google.appengine.ext.vendor, that just does this for me.
I will note that the way I do this in a production app is to use a requirements.txt file containing my packages (eg: flask), and use a build process to invoke pip on this, sorting out these third party dependencies in an orderly fashion. But this method will do fine for this app.
Note: appengine_config.py is some kind of magic bean. I think it runs once when a new instance is initialised for one of your app versions, but I don’t know exactly. It might be something to experiment on a bit another day.
So I’ve added appengine_config.py as above, and installed flask with the pip command. Here’s my app folder now:
Ooh, that installed a lot of stuff in the lib folder. Cool bananas.
Setting up the yamls
The next bit of the guide says to set up app.yaml. I’ll set up app.yaml, queue.yaml, and background.yaml , like this:
app.yaml
runtime: python27
api_version: 1
threadsafe: true
automatic_scaling:
max_idle_instances: 2
handlers:
- url: /.*
script: main.appbuiltins:
- deferred: on
background.yaml
runtime: python27
api_version: 1
threadsafe: true
service: background
automatic_scaling:
max_idle_instances: 2handlers:
- url: /.*
script: main.appbuiltins:
- deferred: on
queue.yaml
queue:
- name: default
rate: 100/s
- name: background
target: background
rate: 100/sSee Python App Engine 2017: Prioritisation of tasks using Services for details on why I’ve done this.
Creating the basic pages
Ok. The next bit of the guide is about setting up handler. I need two GET handlers, one for the switchboard, one for the report page.
I’ll put the switchboard at /, and the report page at /report.
For the first shot at this, I’ll make both these pages not really do anything.
Note: To get Eclipse to play nicely with the vendored libraries, I’ve added the lib folder to the project’s PYTHONPATH, like this:
Ok, here’s my main.py:
import loggingfrom flask import Flaskapp = Flask(__name__)@app.route('/')
def switchboard():
return 'Switchboard'@app.route('/report')
def report():
return 'Report'@app.errorhandler(500)
def server_error(e):
# Log the error and stacktrace.
logging.exception('An error occurred during a request.')
return 'An internal error occurred.', 500
And let’s run it:
That works. Now, let’s refactor this a bit to put switchboard and report in their own files.
main.py
import loggingfrom flask import Flaskapp = Flask(__name__)from handlers.switchboard import get_switchboard
from handlers.report import get_reportget_switchboard(app)
get_report(app)@app.errorhandler(500)
def server_error(e):
# Log the error and stacktrace.
logging.exception('An error occurred during a request.')
return 'An internal error occurred.', 500
I’ll throw an empty __init__.py file into the handlers folder, so it works as a package. Then,
switchboard.py
def get_switchboard(app):
@app.route('/')
def switchboard():
return 'Switchboard' return switchboard
report.py
def get_report(app):
@app.route('/report')
def report():
return 'Report' return report
I don’t know if that’s recommended, but it seems like a good structure to me. Ok.
Switchboard
Here I want to list the experiments, one button per experiment.
I don’t have any experiments yet, so let’s make a hello world experiment, which creates a hello world object that we can report on.
Model
By object, I mean a hello world model object. I’ll pop it in a model folder.
Like this:
model/helloworld.py
from google.appengine.ext import ndbclass HelloWorld(ndb.model.Model):
stored = ndb.DateTimeProperty(auto_now_add=True)
updated = ndb.DateTimeProperty(auto_now=True) def to_dict(self):
return {
"key": self.key.urlsafe() if self.key else None,
"stored": str(self.stored) if self.stored else None,
"updated": str(self.updated) if self.stored else None
}
I don’t want to dive into ndb model objects here. The gist is that this class is something you can store to the Google Cloud Platform Datastore, and load back up. It’s got two persistent attributes, stored and updated, which are automatically set to the time the object was first stored, and the time the object was last updated, respectively. It also has a Key, which is implied.
I’ve added a to_dict() method which returns a JSON compatible dictionary full of attributes in the object. We’ll need that for report.
Experiments
I need a way to define an experiment. Let’s do it in a functional way.
I need an experiment to have
- a displayable name, and
- a function that takes no arguments, actually kicks off the experiment, and returns the key to a datastore object which I can report on.
Here’s a HelloWorld experiment.
experiments/helloworld.py
from model.helloworld import HelloWorlddef HelloWorldExperiment():
def Go():
lhw = HelloWorld()
lhw.put()
return lhw.key
return "Hello World", Go
The Go method actually performs the experiment. It creates and stores a HelloWorld model object, and returns its key.
HelloWorldExperiment returns a pair; the name of the experiment, and the Go function.
Great. Now how do we do the switchboard?
Switchboard
Here’s an updated switchboard function:
handlers/switchboard.py
from flask import render_templatefrom experiments.helloworld import HelloWorldExperimentdef get_switchboard(app):
@app.route('/')
def switchboard():
experiments = [
HelloWorldExperiment()
]
return render_template(
"switchboard.html",
experiments = experiments
) return switchboard
There are a few new concepts here.
In the inner switchboard function, I’m constructing a list of experiments (just what is returned byHelloWorldExperiment at the moment, ie: the pair of name and function), then passing it to render_template. This magically finds templates/switchboard.html, and renders it using jinja2 with the array experiments in the context.
Remember that HelloWorldExperiment() returns a pair of name and function? This means experiments is a list of these pairs.
So now we need a template that does something useful with this array.
templates/switchboard.html
<html>
<head>
<title>Experiments</title>
</head>
<body>
<h1>Experiments</h1>
{% for ex in experiments %}
<p>
<a href="report?key={{ex[1]().urlsafe()}}">{{ex[0]}}</a>
</p>
{% endfor %}
</body>
</html>This iterates through the experiments, showing each one as a link with the text being the experiment name, and the href being reports?key=<key>. <key> is the key returned from calling the experiment function; the urlsafe() method turns it into something we can put on the querystring.
Lastly, let’s throw in a simple update to the report handler which will just display the key.
handlers/report.py
from google.appengine.ext import ndb
from flask import requestdef get_report(app):
@app.route('/report')
def report():
keystr = request.args.get('key')
return str(ndb.Key(urlsafe=keystr))
return report
Let’s take a look:
Let’s click Hello World:
That’s working; you’re looking at the string representation of the ndb Key for the model object that we created. Nice!
Ok, let’s do the report page in earnest.
Report
Alright. This is going to work as follows:
- get the key
- load the object for the key
- call to_dict() on it
- convert that to a string representation
- return that as the report.
Here it is:
handlers/report.py
from google.appengine.ext import ndb
from flask import request, render_template
import jsondef get_report(app):
@app.route('/report')
def report():
keystr = request.args.get('key')
key = ndb.Key(urlsafe=keystr)
obj = key.get()
return render_template(
"report.html",
objjson = json.dumps(
obj.to_dict(),
indent=2,
sort_keys=True
),
keystr = keystr
) return report
You can see it gets the string representation of the key, constructs the key, loads the object using the key, then renders the template, passing a json string representation of the object (this uses to_dict() on the model object).
What about the template?
<html>
<head>
<title>Report for {{keystr}}</title>
<link rel="stylesheet" type="text/css" href="/static/style.css">
</head>
<body>
<h1>Report for {{keystr}}</h1>
<h3><a href="/">< experiments</a></h3>
<pre>{{objjson|safe}}</pre>
</body>
</html>Simple template; just grabs the json and throws it onto the page.
Let’s refresh the report page and see what it looks like now:
That’s pretty good!
I’m going to upload this to app_engine:
and give it a shot…
Success!
—
Ok, that’s a basic (!) framework I can use for experiments on App Engine in future articles. Flask was pretty awesome, ey? That’s some clean code!
btw, here’s the repo:
Thanks for reading this.
