GCP: Spring To Production With App Engine, Cloud Build And GitHub

We are using this tutorial to walk through getting an Inventory-Management-Rest app through a CI/CD (continuous integration and continuous delivery) pipeline into production. Our goal is to build and deploy on a Google Compute Platform (GCP) App Engine flex environment.

We will be using Spring, Cloud SQL, App Engine, Cloud Secret Manager, Cloud Build and GitHub.

Spring Cloud

Spring Cloud aims to shorten the code length and provide you with the easiest way to develop a web application. With annotation configuration and default codes, Spring Cloud shortens the time involved in developing an application. It helps create a stand-alone application with less or almost zero-configuration.

Cloud SQL

Cloud SQL is a fully managed relational db service for MySQL, PostgreSQL, and SQL Server, run by Google and thus offering easy integration with existing apps and Google Cloud services.

App Engine

App Engine is a platform-as-a-service for developing and hosting web applications at scale on Google-managed infrastructure. It allows you to choose from several popular languages, libraries, and frameworks to develop and deploy your apps, including Java with Spring. Applications are sandboxed and deployed across Google infrastructure, taking away the need for environment configuration and management, across the application lifecycle. Makes it easy, to write scalable applications.

Cloud Secret Manager

Secret Manager is a secure and convenient storage system for API keys, passwords, certificates, and other sensitive data. Secret Manager provides a central place and single source of truth to manage, access, and audit secrets across Google Cloud, or anywhere else for that matter.

This is a relatively new feature by Google (as of May 2020) that’s meant to close the gap around the GCP ecosystem that has existed for quite a long-time. Infact, our deployment here, provides a good use case:

How do you easily manage credentials, passwords, secrets, api keys and other sensitive data, while hosting your application on a public repository (aka GitHub) with a CI/CD pipeline, that requires the sensitive data to successfully build?

I talk about secret manager a bit more on this article.

Cloud Build

Cloud Build is a service that executes your builds on Google Cloud Platform infrastructure. Cloud Build can import source code from Cloud Storage, Cloud Source Repositories, GitHub, or Bitbucket, execute a build to your specifications, and produce artifacts such as Docker containers or Java archives.

Github

Need I say more….

Prerequisites
1. Create a project in the Google Cloud Platform Console.
2. Enable billing for your project.
3. Install the Google Cloud SDK.
4. Install maven
5. Create a GitHub account

Getting Started

1. Start by initializing the Cloud SDK, create an App Engine application, and authorize the Cloud SDK to use GCP APIs in your local environment:

gcloud init
gcloud app create
gcloud auth application-default login

Set up Cloud SQL

1. Enable the Cloud SQL API.
2. Create a Cloud SQL (MySQL) instance and set the root user password following these instructions.

gcloud sql instances create test-instance-inventory-management --tier=db-n1-standard-1 --region=us-central1

3. Set the password for the “root@%” MySQL user

gcloud sql users set-password root --host=% --instance test-instance-inventory-management --password [PASSWORD]

Make sure you replace [PASSWORD] with your own password

4. Setup inventory database.

gcloud sql databases create inventory --instance=test-instance-inventory-management

5. Get the connectionName of the instance in the format project-id:zone-id:instance-id:

gcloud sql instances describe test-instance-inventory-management | grep connectionName

Now lets test out our Cloud Sql Database and see if we are able to use this database. We’ll start by cloning the project and testing locally

git clone https://github.com/kioie/InventoryManagement.git
cd InventoryManagement/

We’ll quickly update our application-mysql.properties file. We will replace the instance-connection-name, database-name, username and password.

Note: The values you will find when you first open this file have been designed for a secret manager connection which we will discuss about shortly

Update src/main/resources/application-mysql.properties:

#CLOUD-SQL-CONFIGURATIONS
spring.cloud.appId=sample-gcp-project
spring.cloud.gcp.sql.instance-connection-name=sample-gcp-project-277704:us-central1:test-instance-inventory-management
spring.cloud.gcp.sql.database-name=inventory
##SQL DB USERNAME/PASSWORD
spring.datasource.username=root
spring.datasource.password=xxxxx

Note: Replace `xxxxx` with your preconfigured db instance password

Start the Spring Boot application

mvn spring-boot:run

Test your application

curl http://localhost:8080/inventory/

Set up Secret Manager

1. Enable the Secret Manager API

You will need to grant the application access
- Go to IAM & Admin page
- Click the Project selector drop-down list at the top of the page.
- On the Select from dialog that appears, select the organization for which you want to enable Secret Manager.
- On the IAM page, next to the app engine service account, click Edit.
- On the Edit permissions panel that appears, add the necessary roles.
- Click Add another role. Select Secret Manager Admin.
- When you’re finished adding roles, click Save.

2. Create new secrets for our datasource configuration file.

echo -n “sample-gcp-project-277704:us-central1:test-instance-inventory-management” | gcloud secrets create spring_cloud_gcp_sql_instance_connection_name — replication-policy=”automatic” — data-file=-
 
echo -n “inventory” | gcloud secrets create spring_cloud_gcp_sql_database_name — replication-policy=”automatic” — data-file=-
 
echo -n “root” | gcloud secrets create spring_datasource_username — replication-policy=”automatic” — data-file=-
 
echo -n “test123” | gcloud secrets create spring_datasource_password — replication-policy=”automatic” — data-file=-

Note: Remember to use your own credentials here for spring_cloud_gcp_sql_instance_connection_name and spring_datasource_password

Confirm your secrets have been created by running gcloud secrets list

3. We now replace the values in the application-mysql.properties file with the secrets url. For this step, you will need the fully-qualified name of the secret as defined on GCP.

gcloud secrets describe spring_cloud_gcp_sql_instance_connection_name | grep name
 
gcloud secrets describe spring_cloud_gcp_sql_database_name | grep name
gcloud secrets describe spring_datasource_username | grep name
gcloud secrets describe spring_datasource_password | grep name

We will now use this names to create a secret manager url. The url will use the format ${sm://FULLY-QUALIFIED-NAME} where FULLY-QUALIFIED-NAME is as retrieved above.

So we update src/main/resources/application-mysql.properties:

#CLOUD-SQL-CONFIGURATIONS
spring.cloud.appId=sample-gcp-project
spring.cloud.gcp.sql.instance-connection-name=${sm://projects/.../secrets/spring_cloud_gcp_sql_instance_connection_name}
spring.cloud.gcp.sql.database-name=${sm://projects/.../secrets/spring_cloud_gcp_sql_database_name}
##SQL DB USERNAME/PASSWORD
spring.datasource.username=${sm://projects/.../secrets/spring_datasource_username}
spring.datasource.password=${sm://projects/.../secrets/spring_datasource_password}

Restart the Spring Boot application:

mvn spring-boot:run

Test your application:

curl http://localhost:8080/inventory/1

Set up GitHub repository with source files

1. Create a GitHub account if you don’t have one already.
2. Enable the Cloud Build API in the target Cloud project.
3. Fork this repository before continuing. This step is important as you will be pushing your changes to this repo
   a. Navigate to https://github.com/kioie/InventoryManagement
   b. On the top-right corner of the page, click Fork.

Fork Project

4. Install the Google Cloud Build App on Github. You can follow this instructions here.

Make sure you select the kioie/InventoryManagement repository fork as the repository you are connecting to, otherwise this build will not work.

5. In the Google Cloud Console, open the Cloud Build Build triggers page.

Make sure to delete all triggers created, that may be related to this project before moving to the next step.

6. Select your Google Cloud project and click Open.

7. Click Create Trigger.

8. Fill out the options

• Required. In the Name field, enter a name
• Optional. In the Description field, enter a brief description of how the trigger will work
• Under Event, select Push to a branch
• In the Source drop-down list, select kioie/InventoryManagement repository
  Note: If this repository does not appear, click the Connect New Repository button and connect your repo on GitHub, then return to step5.
• Under Branch, enter ^master$
• Under Build Configuration select Cloud Build configuration file (YAML or JSON)
  For the Cloud Build configuration file location enter cloudbuild.yaml
  Note: Do not add an extra /
• Click create

9. Under your active triggers, you should be able to see your newly created trigger.

Some Brief information

As a requirement for the Google Cloud Build app, your repository must contain either a Dockerfile or a cloudbuild.yaml file to be able successfully configure your build.

Dockerfile is generally used for building Docker containers. Incase you are using Cloud Build for Docker Builds, you will require a Dockerfile. This tutorial is for an App Engine build, but you will notice that the sample repo contains a Dockerfile. This is because the cloud build contains a bonus step that completes by creating a container artifact, although this is not necessary for this tutorial.

cloudbuild.yaml is the config file for Cloud Build. You use a cloudbuild.yaml in the following scenarios:

• When using Cloud Build app for non-Docker builds.
• If you wish to fine-tune your Docker builds, you can provide a cloudbuild.yaml in addition to the Dockerfile. If your repository contains a Dockerfile and a cloudbuild.yaml, the Google Cloud Build app will use the cloudbuild.yaml to configure the builds.

Set up App Engine

We will require app engine for our deployment. In the end, our application will be deployed and accessed on https://YOUR_PROJECT_ID.appspot.com

1. The pom.xml already contains configuration for projectId and version. We are going to change this to reflect the current project ID.
   <deploy.projectId>sample-gcp-project-276208</deploy.projectId>
2. Enable App Engine Admin API
3. Enable App Engine Flexible API
4. We will need to give more permission to the cloud build service account

You will need to grant cloudbuild service account, admin access to Secret Manager, App Engine and Cloud Sql.

• Go to IAM & Admin page
• Click the Project selector drop-down list at the top of the page and select the current project organization.
• On the IAM page, next to the cloud build service account, (not to be confused with the cloud build service agent )click Edit (or the pencil button).
• On the Edit permissions panel that appears, add the necessary roles.
• Click Add another role. And add the specific role.
  We will be adding 3 roles:
  — App Engine Admin
  — Cloud SQL Admin
  — Secret Manager Admin
• When you’re finished adding roles, click Save.

The final permission list should look something like this

Permissions list

We now come to the final step of triggering a cloud build. We will push our updated code-base to GitHub.

Push to GitHub and trigger a build

1. Add your remote GitHub fork repo as your upstream repo

git remote add upstream https://github.com/YOUR_ACCOUNT_NAME/InventoryManagement
git remote -vv

2. Commit your changes

git add .
git commit

3. Push upstream

git push upstream master

4. This should automatically trigger your build on GCP cloud build. You can check the status of your build using the below command

gcloud builds list

5. Hopefully your build completes with a success result, if so, you can fetch the url of the app with the command

gcloud app browse

6. We can now test out our endpoints

curl https://sample-gcp-project-277704.uc.r.appspot.com/inventory
curl https://sample-gcp-project-277704.uc.r.appspot.com/inventory/1
curl https://sample-gcp-project-277704.uc.r.appspot.com/inventory/2

If everything went well, you now have an app running on GCP App Engine, deployed using Cloud Build, and hosted on GitHub, and using Secret Manager to maintain your secrets.