Deploying Kotlin/Java applications to Google App Engine Standard with CircleCI

Let’s say that you have decided to use Google App Engine Standard for your new service, which is written in some JVM-based language (Kotlin, Java, …) and after some fiddling, it is now happily running on your computer.
You might also have deployed it to App Engine already using mvn appengine:deploy, and it seems to be working fine in its natural habitat as well.

Thinking about Continuous Integration — especially if you have your code on GitHub — you might end up using CircleCI.
In their documentation they name a handful of deployment targets alongside Google App Engine, but what makes GAE different is that you need the Google Cloud SDK not just for deploying, but also for building your deployable artifact.

Installing the Cloud SDK

Since we are talking about deploying a JVM based project in 2018, the Docker image used for building the project should have at least JDK 8 installed — alongside with the Google Cloud SDK.

Google does provide a Docker image that has the Cloud SDK installed, but at the time of writing, that one does not include a JDK — at least not officially. Even though a quick trial did reveal that there was a JDK installed, that was only JDK 7 — and lacking any official mentions, there was no guarantee either that it would be kept there.

So, there are two options — either use a JDK-enabled base image and install the Cloud SDK, or start from the Cloud SDK and install a JDK.

CircleCI gives you the possibility to cache directories, and then in subsequent builds, reinstate their contents, thus saving time by not repeating the steps required to produce those contents.
Caching node_modules after npm install is a typical use-case.
Given that the list of directories to cache is manually set, the more self-contained your to-be-cached content is the better.

Installing a JDK would likely have meant running something along the lines of apt-get install, which spreads installations all over the place. Installing the Cloud SDK, however, is possible by downloading and extracting an archive inside a directory of your choice.
This made me start from a JDK image (openjdk-8) and write a script that downloads and initializes the Cloud SDK as part of the build process:

The script checks first if the SDK is already present at the given location by running gcloud version — if that succeeds, then the SDK download is skipped. Otherwise, the SDK is downloaded and extracted under the current user’s home directory.
The next step is to install the App Engine component, which is taken care of by running gcloud components install app-engine-java.
The final step is to initialize the SDK — in other words, set a service account and set the current project.

The method for initialization — setting and activating service account credentials — is the same as presented in the CircleCI guides.
In short — the Cloud SDK expects a JSON file that is passed in to the build job via an environment variable that contains the file’s contents in base-64 encoded format.

To keep the installation up-to-date, the script could include runninggcloud components update — this updates the SDK to the latest stable version.
Based on my experience, latest is not always the greatest of the Cloud SDK — at least for now — so you can decide if you want to stick with something that works, or try out what’s new.
Caching the installation — the method described further below — would also have to be adjusted if the Cloud SDK was updated from inside this script.

CircleCI build configuration

Here is an example CircleCI configuration that uses the script above:

Usual steps (caching the build’s dependencies, etc) are omitted for clarity

The steps, one-by-one:

1. Checkout the code
2. Restore the Cloud SDK installation. The cache key is the checksum of the installation script, so that when it changes (e.g you change the version to download) the cache becomes invalid, and nothing is restored.
3. Run the Cloud SDK installation script. This will pick up the restored installation (if any) and if found, skip the download part — but will execute the initialization regardless.
4. Save the installation in the cache. This is using the same key as the restoration step, otherwise caching would not work.
5. Run the Maven deployment target (which also runs the tests & assembles the artifacts to deploy)

Bonus: continuous delivery

Making use of the CircleCI approval steps, it does not take much additional effort to add multiple deployment jobs, separated by manual approvals.
But first — assuming your test/QA/production services live under different GCP projects — the installation script must be extended:

Instead of deploying to one GCP project, now it is possible to deploy to multiple ones. The difference between this version and the previous one is that instead of relying directly on CircleCI environment variables, the script sets up intermediate variables that source their values based on which environment is selected (i.e passed in as first argument to the script).

Let’s say that you have two projects, one for test and another for production. Then you will need the following environment variables set:

• GCLOUD_PROJECT_TEST — GCP project name for the test environment
• GCLOUD_SERVICE_KEY_TEST — base-64 encoded contents of the service account key JSON for the test GCP project
• GCLOUD_PROJECT_PROD — GCP project name for the prod environment
• GCLOUD_SERVICE_KEY_PROD — base-64 encoded contents of the service account key JSON for the prod GCP project

With these in place, the CircleCI configuration can be the following:

Usual steps (caching the build’s dependencies, etc) are omitted for clarity

The list of changes:

• The build job does not deploy — it just executes appengine:stage so that the directory containing the artifacts is prepared, and then saved to the CircleCI workspace.
• There are additional jobs for deployment: deploy-to-test and deploy-to-prod. These both mount the saved workspace from the build job, so that they have access to the staged artifact. Their main task is to execute gcloud app deploy inside the stage directory.
  Note that the SDK installation step differs between these two jobs; the one for test has no parameters (and defaults to using the *_TEST environment variables), while for prod the first parameter is PROD, putting the *_PROD variables in use.
• There is a workflow defined for the master branch that executes the build and deploy-to-test jobs, and then if approved manually, runs the deploy-to-prod job as well.

By using this setup, your project will get automatically built & deployed to test, then you can choose to continue to production by approving it in CircleCI.

It is of course possible to extend the workflow by adding even more stages — e.g a QA environment.

https://circleci.com/docs/2.0/deployment-integrations/

https://cloud.google.com/appengine/docs/standard/java/