App Engine standard in Go: en route to 1.11
TL;DR migrating to the new runtime is a good idea, give it a try!
I’ve been running a website on the App Engine standard go runtime for years (since ~2013), and I’ve always been thankful for how well it is managed by the cloud provider:
- What I deploy simply keeps running.
- I never care about OS security patches or database upgrades.
- Scaling up and down is automatic.
- The traffic (~6000 sessions/month) mainly fits within the generous free tier. I pay nothing for the server instances, for the Datastore and Memcache, and that’s basically all I need.
Serverless, before “serverless” was even a word!
The new Go 1.11 runtime for App Engine was announced in October 2018. Let’s have a look!
Why would I want to migrate to the new runtime Go 1.11 (beta)?
Novelties in the language and in the core library
Novelties in the App Engine runtime
- Old restrictions have been removed: any package can now be imported and vendored ; the filesystem is accessible (in
/tmp) ; and regular HTTP clients can be used.
- Apps are no more tied to App Engine. You can write a go web server in an idiomatic way and host it anywhere.
- Go modules are supported.
The changes needed in the source code of the app to use the runtime “go111” are described in the official documentation Migrating your App Engine app from Go 1.9 to Go 1.11 .
Note that billing must be enabled in the GCP project in order to use the beta runtime. To avoid disturbing your existing prod, you may experiment and deploy to an alternative app version, or to an alternative GCP project.
Cloud vendors don’t have an easy job maintaining all the stack of hardware and software required to host customer apps. Things rot! Ecosystems evolve, diverge and converge. Security hazards are patched and yesterday’s best practices get deprecated.
The App Engine-specific APIs, i.e. all imports starting with
google.golang/org/appengine , are approaching end-of-life. Instead, it is recommended to use Google Cloud client libraries, starting with
cloud.google.com/go, wherever possible.
The bad news is that not all App Engine APIs have a straightforward replacement available, yet.
The good news is that all App Engine APIs still work in runtime go111.
Future GAE runtimes (Go 1.12, etc.) won’t support the legacy APIs. However, go111 will still be supported in 2019 and 2020, so next time you redeploy, consider switching to go111. The transition to more modern and decoupled libraries can be done later, gradually.
If you’re still using
goapp (from the “original” App Engine SDK for Go) to deploy to prod (to the cloud), forget about it.
Now I deploy with
$ gcloud app deploy
This last command assumes that my
gcloud is already configured with my project
programming-idioms, and that I want to deploy a “current version”, i.e. a new app version that will serve 100% of the incoming requests.
The project can be set as a property of the active configuration in gcloud:
$ gcloud config set project programming-idioms
Or I can specify it in the deploy command:
$ gcloud --project programming-idioms app deploy
Note that the values for
application(GCP project) and
versionare no longer in file
If you’re still using
goapp to start a local server, forget about it.
A modern GAE app written in go is basically a regular web server, that can be compiled and started locally like any go program. That’s great in theory, and I recommend it for writing a new app from scratch.
However if you’re transitioning gradually (like me) and still have adherence to a few appengine old habits: static JS/CSS files, datastore, memcache… then I suggest this command instead:
$ dev_appserver.py .
The original source layout of my app looks like this
Folder pigae contains all the Go source files of the web backend
- All sources are in
- There is no main func.
- There is no main package.
Folder static contains all the JS, CSS, PNG files
- They are marked as
app.yaml, in order to leverage CDN-like performance and free egress.
- They enjoy a long-time browser cache policy, thanks to a little trick consisting in changing the resource path (e.g. …/20171211_default/… ) whenever a change to at least 1 static resource is deployed.
Folder template contains all the HTML files parsed with package html/template: all pages are rendered server-side.
index.yaml contains the Datastore indexes definitions.
Following the migration doc
The basic elements are straightforward to change:
If you have handlers with
static_dir in your yaml, beware of this inconspicuous bizarre rule in the app.yaml reference:
I don’t really want to know what’s going on here, so I just comply:
Folders and packages
It is now mandatory that my server starts from
package main. However:
pigaecan’t cohabit in the same folder
- If I just replace
package mainin all source files, then the app current folder will be “pigae”, which does not contain “template”, needed at runtime.
- I can either lift down the folder “template”:
- Or lift up the go source files:
Explicit server startup
Because of the dependencies that I still have (at this point) to the App Engine-specific APIs, I can’t really call
http.ListenAndServe and have everything work out-of-the-box. Instead, I import
google.golang.org/appengine and explicitly call
Voilà, I can now push the adapted version to GAE and it works!
I’m very happy that Google engineers decided to keep the legacy APIs working for this new runtime go111. It’s a lot of thankless work and probably includes a few hacks to have new environments cohabit and communicate with old systems.
Also, it feels necessary for apps like mine which heavily rely on Datastore, Memcache, Delayed tasks, Text search.
In follow-up posts, we’ll explore
Moving away from Go AppEngine-specific APIs
- and more
Benefits of the second-generation runtime