App Engine standard in Go: en route to 1.11

Val Deleplace
Dec 21, 2018 · 6 min read

TL;DR migrating to the new runtime is a good idea, give it a try!

I’ve been running a website on the App Engine standard go runtime for years (since ~2013), and I’ve always been thankful for how well it is managed by the cloud provider:

  • What I deploy simply keeps running.
  • I never care about OS security patches or database upgrades.
  • Scaling up and down is automatic.
  • The traffic (~6000 sessions/month) mainly fits within the generous free tier. I pay nothing for the server instances, for the Datastore and Memcache, and that’s basically all I need.

Serverless, before “serverless” was even a word!

Programming-Idioms, written in Go, hosted on App Engine

The new Go 1.11 runtime for App Engine was announced in October 2018. Let’s have a look!

Why would I want to migrate to the new runtime Go 1.11 (beta)?

Novelties in the language and in the core library

I like the TL;DR of these parts of the Go 1.10 and Go 1.11 Release Notes:

A very stable language

Novelties in the App Engine runtime

For the first time, Go apps run in a “second-generation runtime” with a gVisor sandbox. For the developer, this means concretely that:

  • Old restrictions have been removed: any package can now be imported and vendored ; the filesystem is accessible (in /tmp) ; and regular HTTP clients can be used.
  • Apps are no more tied to App Engine. You can write a go web server in an idiomatic way and host it anywhere.
  • Go modules are supported.

The changes needed in the source code of the app to use the runtime “go111” are described in the official documentation Migrating your App Engine app from Go 1.9 to Go 1.11 .

Note that billing must be enabled in the GCP project in order to use the beta runtime. To avoid disturbing your existing prod, you may experiment and deploy to an alternative app version, or to an alternative GCP project.

Service libraries

Cloud vendors don’t have an easy job maintaining all the stack of hardware and software required to host customer apps. Things rot! Ecosystems evolve, diverge and converge. Security hazards are patched and yesterday’s best practices get deprecated.

The App Engine-specific APIs, i.e. all imports starting with google.golang/org/appengine , are approaching end-of-life. Instead, it is recommended to use Google Cloud client libraries, starting with cloud.google.com/go, wherever possible.

The bad news is that not all App Engine APIs have a straightforward replacement available, yet.

The good news is that all App Engine APIs still work in runtime go111.

Future GAE runtimes (Go 1.12, etc.) won’t support the legacy APIs. However, go111 will still be supported in 2019 and 2020, so next time you redeploy, consider switching to go111. The transition to more modern and decoupled libraries can be done later, gradually.

Deployment

If you’re still using goapp (from the “original” App Engine SDK for Go) to deploy to prod (to the cloud), forget about it.

Now I deploy with

$ gcloud app deploy

This last command assumes that my gcloud is already configured with my project programming-idioms, and that I want to deploy a “current version”, i.e. a new app version that will serve 100% of the incoming requests.

The project can be set as a property of the active configuration in gcloud:

$ gcloud config set project programming-idioms

Or I can specify it in the deploy command:

$ gcloud --project programming-idioms app deploy

Note that the values for application(GCP project) and versionare no longer in fileapp.yaml.

Local development

If you’re still using goapp to start a local server, forget about it.

A modern GAE app written in go is basically a regular web server, that can be compiled and started locally like any go program. That’s great in theory, and I recommend it for writing a new app from scratch.

However if you’re transitioning gradually (like me) and still have adherence to a few appengine old habits: static JS/CSS files, datastore, memcache… then I suggest this command instead:

$ dev_appserver.py .

Old structure

The original source layout of my app looks like this

where:

Folder pigae contains all the Go source files of the web backend

  • All sources are in package pigae.
  • There is no main func.
  • There is no main package.

Folder static contains all the JS, CSS, PNG files

  • They are marked as static_files in app.yaml, in order to leverage CDN-like performance and free egress.
  • They enjoy a long-time browser cache policy, thanks to a little trick consisting in changing the resource path (e.g. …/20171211_default/… ) whenever a change to at least 1 static resource is deployed.

Folder template contains all the HTML files parsed with package html/template: all pages are rendered server-side.

index.yaml contains the Datastore indexes definitions.

Refactoring

Following the migration doc

app.yaml

The basic elements are straightforward to change:

If you have handlers with static_files or static_dir in your yaml, beware of this inconspicuous bizarre rule in the app.yaml reference:

I don’t really want to know what’s going on here, so I just comply:

Folders and packages

It is now mandatory that my server starts from package main. However:

  • Packages main and pigae can’t cohabit in the same folder
  • If I just replace package piage with package main in all source files, then the app current folder will be “pigae”, which does not contain “template”, needed at runtime.
  • I can either lift down the folder “template”:
  • Or lift up the go source files:

Explicit server startup

Because of the dependencies that I still have (at this point) to the App Engine-specific APIs, I can’t really call http.ListenAndServe and have everything work out-of-the-box. Instead, I import google.golang.org/appengine and explicitly call appengine.Main() .

Voilà, I can now push the adapted version to GAE and it works!


I’m very happy that Google engineers decided to keep the legacy APIs working for this new runtime go111. It’s a lot of thankless work and probably includes a few hacks to have new environments cohabit and communicate with old systems.

Also, it feels necessary for apps like mine which heavily rely on Datastore, Memcache, Delayed tasks, Text search.

In follow-up posts, we’ll explore

Moving away from Go AppEngine-specific APIs

  • Context
  • Logs
  • Datastore
  • Memcache
  • and more

Benefits of the second-generation runtime

  • Performance
  • Concurrency
  • Tracing

Val Deleplace

Written by

Engineer on cloudy things @Google. Opinions my own. Twitter @val_deleplace