GCP: Spring To Production With App Engine, Cloud Build And GitHub
We are using this tutorial to walk through getting an Inventory-Management-Rest app through a CI/CD (continuous integration and continuous delivery) pipeline into production. Our goal is to build and deploy on a Google Compute Platform (GCP) App Engine flex environment.
We will be using Spring, Cloud SQL, App Engine, Cloud Secret Manager, Cloud Build and GitHub.
Spring Cloud
Spring Cloud aims to shorten the code length and provide you with the easiest way to develop a web application. With annotation configuration and default codes, Spring Cloud shortens the time involved in developing an application. It helps create a stand-alone application with less or almost zero-configuration.
Cloud SQL
Cloud SQL is a fully managed relational db service for MySQL, PostgreSQL, and SQL Server, run by Google and thus offering easy integration with existing apps and Google Cloud services.
App Engine
App Engine is a platform-as-a-service for developing and hosting web applications at scale on Google-managed infrastructure. It allows you to choose from several popular languages, libraries, and frameworks to develop and deploy your apps, including Java with Spring. Applications are sandboxed and deployed across Google infrastructure, taking away the need for environment configuration and management, across the application lifecycle. Makes it easy, to write scalable applications.
Cloud Secret Manager
Secret Manager is a secure and convenient storage system for API keys, passwords, certificates, and other sensitive data. Secret Manager provides a central place and single source of truth to manage, access, and audit secrets across Google Cloud, or anywhere else for that matter.
This is a relatively new feature by Google (as of May 2020) that’s meant to close the gap around the GCP ecosystem that has existed for quite a long-time. Infact, our deployment here, provides a good use case:
How do you easily manage credentials, passwords, secrets, api keys and other sensitive data, while hosting your application on a public repository (aka GitHub) with a CI/CD pipeline, that requires the sensitive data to successfully build?
I talk about secret manager a bit more on this article.
Cloud Build
Cloud Build is a service that executes your builds on Google Cloud Platform infrastructure. Cloud Build can import source code from Cloud Storage, Cloud Source Repositories, GitHub, or Bitbucket, execute a build to your specifications, and produce artifacts such as Docker containers or Java archives.
Github
Need I say more….
Prerequisites
1. Create a project in the Google Cloud Platform Console.
2. Enable billing for your project.
3. Install the Google Cloud SDK.
4. Install maven
5. Create a GitHub account
Getting Started
1. Start by initializing the Cloud SDK, create an App Engine application, and authorize the Cloud SDK to use GCP APIs in your local environment:
gcloud init
gcloud app create
gcloud auth application-default login
Set up Cloud SQL
1. Enable the Cloud SQL API.
2. Create a Cloud SQL (MySQL) instance and set the root user password following these instructions.
gcloud sql instances create test-instance-inventory-management --tier=db-n1-standard-1 --region=us-central1
3. Set the password for the “root@%” MySQL user
gcloud sql users set-password root --host=% --instance test-instance-inventory-management --password [PASSWORD]
Make sure you replace [PASSWORD]
with your own password
4. Setup inventory
database.
gcloud sql databases create inventory --instance=test-instance-inventory-management
5. Get the connectionName
of the instance in the format project-id:zone-id:instance-id
:
gcloud sql instances describe test-instance-inventory-management | grep connectionName
Now lets test out our Cloud Sql Database and see if we are able to use this database. We’ll start by cloning the project and testing locally
git clone https://github.com/kioie/InventoryManagement.git
cd InventoryManagement/
We’ll quickly update our application-mysql.properties
file. We will replace the instance-connection-name, database-name, username and password.
Note: The values you will find when you first open this file have been designed for a secret manager connection which we will discuss about shortly
Update src/main/resources/application-mysql.properties
:
#CLOUD-SQL-CONFIGURATIONS
spring.cloud.appId=sample-gcp-project
spring.cloud.gcp.sql.instance-connection-name=sample-gcp-project-277704:us-central1:test-instance-inventory-management
spring.cloud.gcp.sql.database-name=inventory
##SQL DB USERNAME/PASSWORD
spring.datasource.username=root
spring.datasource.password=xxxxx
Note: Replace `xxxxx` with your preconfigured db instance password
Start the Spring Boot application
mvn spring-boot:run
Test your application
curl http://localhost:8080/inventory/
Set up Secret Manager
- Enable the Secret Manager API
You will need to grant the application access
- Go to IAM & Admin page
- Click the Project selector
drop-down list at the top of the page.
- On the Select from
dialog that appears, select the organization for which you want to enable Secret Manager.
- On the IAM
page, next to the app engine service account
, click Edit
.
- On the Edit permissions
panel that appears, add the necessary roles.
- Click Add another role
. Select Secret Manager Admin
.
- When you’re finished adding roles, click Save.
2. Create new secrets for our datasource configuration file.
echo -n “sample-gcp-project-277704:us-central1:test-instance-inventory-management” | gcloud secrets create spring_cloud_gcp_sql_instance_connection_name — replication-policy=”automatic” — data-file=-
echo -n “inventory” | gcloud secrets create spring_cloud_gcp_sql_database_name — replication-policy=”automatic” — data-file=-
echo -n “root” | gcloud secrets create spring_datasource_username — replication-policy=”automatic” — data-file=-
echo -n “test123” | gcloud secrets create spring_datasource_password — replication-policy=”automatic” — data-file=-
Note: Remember to use your own credentials here for spring_cloud_gcp_sql_instance_connection_name
and spring_datasource_password
Confirm your secrets have been created by running gcloud secrets list
3. We now replace the values in the application-mysql.properties
file with the secrets url. For this step, you will need the fully-qualified name of the secret as defined on GCP.
gcloud secrets describe spring_cloud_gcp_sql_instance_connection_name | grep name
gcloud secrets describe spring_cloud_gcp_sql_database_name | grep namegcloud secrets describe spring_datasource_username | grep namegcloud secrets describe spring_datasource_password | grep name
We will now use this names to create a secret manager url. The url will use the format ${sm://FULLY-QUALIFIED-NAME}
where FULLY-QUALIFIED-NAME
is as retrieved above.
So we update src/main/resources/application-mysql.properties:
#CLOUD-SQL-CONFIGURATIONS
spring.cloud.appId=sample-gcp-project
spring.cloud.gcp.sql.instance-connection-name=${sm://projects/.../secrets/spring_cloud_gcp_sql_instance_connection_name}
spring.cloud.gcp.sql.database-name=${sm://projects/.../secrets/spring_cloud_gcp_sql_database_name}
##SQL DB USERNAME/PASSWORD
spring.datasource.username=${sm://projects/.../secrets/spring_datasource_username}
spring.datasource.password=${sm://projects/.../secrets/spring_datasource_password}
Restart the Spring Boot application:
mvn spring-boot:run
Test your application:
curl http://localhost:8080/inventory/1
Set up GitHub repository with source files
- Create a GitHub account if you don’t have one already.
- Enable the Cloud Build API in the target Cloud project.
- Fork this repository before continuing. This step is important as you will be pushing your changes to this repo
a. Navigate to https://github.com/kioie/InventoryManagement
b. On the top-right corner of the page, clickFork
.
4. Install the Google Cloud Build App on Github. You can follow this instructions here.
Make sure you select the kioie/InventoryManagement
repository fork as the repository you are connecting to, otherwise this build will not work.
5. In the Google Cloud Console, open the Cloud Build Build triggers page.
Make sure to delete all triggers created, that may be related to this project before moving to the next step.
6. Select your Google Cloud project and click Open
.
7. Click Create Trigger
.
8. Fill out the options
- Required. In the
Name
field, enter a name - Optional. In the
Description
field, enter a brief description of how the trigger will work - Under
Event
, selectPush to a branch
- In the
Source
drop-down list, selectkioie/InventoryManagement
repository
Note: If this repository does not appear, click theConnect New Repository
button and connect your repo on GitHub, then return to step5. - Under
Branch
, enter^master$
- Under
Build Configuration
selectCloud Build configuration file (YAML or JSON)
For theCloud Build configuration file location
entercloudbuild.yaml
Note: Do not add an extra/
- Click
create
9. Under your active triggers, you should be able to see your newly created trigger.
Some Brief information
As a requirement for the Google Cloud Build app, your repository must contain either a Dockerfile
or a cloudbuild.yaml
file to be able successfully configure your build.
Dockerfile
is generally used for building Docker containers. Incase you are using Cloud Build for Docker Builds, you will require a Dockerfile
. This tutorial is for an App Engine build, but you will notice that the sample repo contains a Dockerfile
. This is because the cloud build contains a bonus step that completes by creating a container artifact, although this is not necessary for this tutorial.
cloudbuild.yaml
is the config file for Cloud Build. You use a cloudbuild.yaml
in the following scenarios:
- When using Cloud Build app for non-Docker builds.
- If you wish to fine-tune your Docker builds, you can provide a
cloudbuild.yaml
in addition to theDockerfile
. If your repository contains aDockerfile
and acloudbuild.yaml
, the Google Cloud Build app will use thecloudbuild.yaml
to configure the builds.
Set up App Engine
We will require app engine for our deployment. In the end, our application will be deployed and accessed on https://YOUR_PROJECT_ID.appspot.com
- The
pom.xml
already contains configuration forprojectId
andversion
. We are going to change this to reflect the current project ID.<deploy.projectId>sample-gcp-project-276208</deploy.projectId>
- Enable App Engine Admin API
- Enable App Engine Flexible API
- We will need to give more permission to the cloud build service account
You will need to grant cloudbuild service account, admin access to Secret Manager, App Engine and Cloud Sql.
- Go to IAM & Admin page
- Click the
Project selector
drop-down list at the top of the page and select the current project organization. - On the IAM page, next to the
cloud build service account
, (not to be confused with thecloud build service agent
)clickEdit
(or the pencil button). - On the
Edit permissions panel
that appears, add the necessary roles. - Click
Add another role
. And add the specific role.
We will be adding 3 roles:
— App Engine Admin
— Cloud SQL Admin
— Secret Manager Admin - When you’re finished adding roles, click
Save
.
The final permission list should look something like this
We now come to the final step of triggering a cloud build. We will push our updated code-base to GitHub.
Push to GitHub and trigger a build
- Add your remote GitHub fork repo as your upstream repo
git remote add upstream https://github.com/YOUR_ACCOUNT_NAME/InventoryManagement
git remote -vv
2. Commit your changes
git add .
git commit
3. Push upstream
git push upstream master
4. This should automatically trigger your build on GCP cloud build. You can check the status of your build using the below command
gcloud builds list
5. Hopefully your build completes with a success result, if so, you can fetch the url of the app with the command
gcloud app browse
6. We can now test out our endpoints
curl https://sample-gcp-project-277704.uc.r.appspot.com/inventory
curl https://sample-gcp-project-277704.uc.r.appspot.com/inventory/1
curl https://sample-gcp-project-277704.uc.r.appspot.com/inventory/2
If everything went well, you now have an app running on GCP App Engine, deployed using Cloud Build, and hosted on GitHub, and using Secret Manager to maintain your secrets.