Building, Testing, and Optimizing Code on Google Cloud

Summary

This post explains the use of Google Cloud Platform tools for building and testing Go code with an application that can be run locally, in unit tests, a plain Docker container, and in App Engine Flex with a custom container. Building, testing, and deployment of the code are driven from Cloud Build. The sample code is a basic but non-trivial app called devflowapp similar in structure to a real-world web application, which may serve as a template for your projects. The code optimization tools Cloud Trace, with OpenCensus, and Cloud Profiler are demonstrated with the app.

Background

Google Cloud Build is a multi-language integration tool for running builds and tests on the Google Cloud Platform. I often find myself collecting commands to test, build, package, and deploy code for daily development work. The nice thing about Cloud Build is that you can format these commands into build config files that are easy to re-run better toshare with others than a long recipe of commands. The simple but non-trivial app provides a convenient platform for tracing and profiling, which is essential to any app for optimizing costs and providing a good user experience.

One of the nice things about App Engine Flex is that it allows you to run regular code that does not have any imports that tie you to App Engine. App Engine Standard has been reducing dependencies but is not quite completely free of them at the time of writing this page. The minimal dependencies in Flex enables you to take advantage of App Engine for the convenience of running in a unified environment and then move your code to somewhere else later, if needed.

Cloud Project Setup

The prerequisites for running the app are described in the GitHub example README file.To perform profiling and tracing you will also need to enable the Stackdriver Profiling API and the Stackdriver Trace API.

Project Directory Layout

Project directory layout is one of the first considerations in beginning a new project. One possible choice is the Go project layout recommendations suggestion. Note that this is not a standard and does not necessarily have the community has coalesced around it. The project layout in this example is

/build

Cloud Build config files

/data

Database configuration files

/deployment

Deployment artifacts

/services

A Go package directory

If you are trying this out in your own workspace you can get the code with the git clone command and change directories to devflowapp.

git clone https://github.com/GoogleCloudPlatform/golang-samples.git cd go/src/github.com/GoogleCloudPlatform/golang-samples/getting-started/devflowapp

Local Development Environment

Follow the steps in the README for running and unit testing the application in a local development environment. The instructions there including setting up a MySQL database for the application using Cloud SQL. If you are using something other than a SQL database you write your own implementation of the MessageService interface. A real messaging application would need a storage system and a notification service. The notification service might be Firebase Cloud Messaging for web and mobile application notification.

Packaging in a Docker Container

The README also describes packaging the app in a Docker container. This activity can enable deploying your app outside your development environment. One choice is deploying to App Engine Flex, which is described in the GitHub example. For many applications, App Engine is the easiest and most cost effective way to serve a web application on the Internet. However, with the custom contain approach followed in the example, you can package the same code to Kubernetes.

Testing and Performance Optimization

Once you have your application up and running in at least one environment, the next steps are to automate end-to-end testing, apply some load, and optimize performance. Optimizing performance can be divided into profiling, which focuses on resource usage, and tracing, which focuses on latency.

Integration and Load Testing

The GitHub example provides an example of integration testing using uses curl with the cb-e2etest.yaml. With a version of the app running on App Engine Flex, we are in a good position to run that with the command.

gcloud builds submit — config build/cb-e2etest.yaml .

Curl is a great tool that can craft complex HTTP requests suitable for automating user access to web sites. The example build file uses curl globbing to invoke the request a number of times in order to generate sufficient data to profile the app. A more sophisticated approach to integration testing would be to use a tool like Selenium WebDriver.

Profiling

Now that we have generate some load on the app, we can look at the performance profile. You can profile the code with Stackdriver Profiler to identify areas that can be optimized. The steps here are adapted from the page Profiling Go Code.

If you have not enabling the Stackdriver Profiler API you can do that with this command:

gcloud services enable cloudprofiler.googleapis.com

To enable Stackdriver Profiler you need to add some code to the devflowapp.go example. That code is provided in this Gist. You may need to use Go 1.10 in the Dockerfile to get this to build because of some changes in dependencies. Then redeploy to App Engine Flex with the command

DB_PASSWORD=[user db password]
gcloud builds submit \
  --substitutions=_DB_PASSWORD=$DB_PASSWORD \
  --config build/cb-deploy.yaml .

as above, and running the integration test, we can look at the profline in the Cloud Console. A screenshot is shown below.

Screenshot of the CPU Profile for BasicApp

After drilling down from main() to ServeHTTP, the handleCheckMessages method can be seen to be consuming quite a lot of time. The reason for this is that there is no bound on the number of messages retrieved. If you run the end-to-end test a sufficiently large number of times manny messages will be inserted in the database causing the resources spent on retrieving message to grow substantially.

Tracing

Stackdriver Cloud Trace is a distributed trace system, including management and presentation of trace data. You application can be instrumented for trace data collection with OpenCensus, an open source framework for monitoring and trace metric collection. Trace data collected with OpenCensus can be managed and viewed with other backends in addition to Stackdriver. The code in the Gist, which includes the code changes necessary for tracing, are adapted from the OpenCensus Stackdriver Codelab.

Make sure that the Stackdriver Trace API is enabled and enable Application Default Credentials for authentication. If you are running on App Engine or GKE then this will be enabled already but you will need it for local development and GCE.

As screenshot of Stackdriver Trace is shown below.

More

For more on Google Cloud Build see the recording of the Cloud Build documentation page. Also see the Go Bookshelf example app for the use of a number of storage options with Go and Building an App with Go for an example of web development with Go on App Engine Standard.

Other development tools that you might want to explore include:

1. Stackdriver Cloud Debugger — for inspecting the state of a running application
2. Go test benchmars — for details on performance benchmarking
3. Gometalinter- for an approach to static code analysis
4. OpenMetrics — standard proposal for metrics collection standard
5. Google Cloud Build community images — community-contributed builders