Feature deployment on GCP

Published in

hipay-tech

6 min readDec 15, 2022

Context

One of the problems of having several developers working on the same project is that we need to ensure a separate testing environment for each of them.

For example, if Johnny is working on a “feature-1” ticket and Laeticia is working on a “feature-2” ticket, it is imperative that the development environment (where each developer’s code is deployed) has a “dev-feature-1” sub-environment and a “dev-feature-2” sub-environment in order to guarantee an exclusive and independent working environment.

This architecture ensures that Johnny will not be impacted by the work of Laeticia.

Breakdown of GCP projects

To implement this mechanism in the reconciliation team at Hipay, we created 4 GCP projects (one per environment):

reco-dev ⇒ development environment
reco-qa ⇒ test environment
reco-stage ⇒ production environment for merchant testing
reco-prod ⇒ production environment

In our case, each GCP project accepts only a certain type of branch (git branch)(¹):

reco-dev (multi-branch) ⇒ develop & feature
reco-qa (multi branch) ⇒ release, tag & hotfix
reco-stage (single branch) ⇒ tag
reco-prod (single branch) ⇒ tag

Production projects (reco-stage & reco-prod) can only have one branch simultaneously, which is the last tag.

Multi-branch projects

We will therefore look at multi-branch projects to understand how to implement the feature deployment mechanism.

To partition a sub-environment (feature, hotfix etc…) in a GCP project (reco-dev or reco-qa), it is necessary that each of them is separated from the other sub-environments.

The deployment consists of several GCP resources that communicate with each others (micro service architecture):

HTTP Cloud Functions
Pubsub Cloud Functions
Pubsub topics
Firestore collections
Cloud Run
Buckets in Cloud Storage
Service Accounts
Schedulers
VPC
etc

Taking the previous example, the Cloud Functions deployed by Johnny for his “feature-1” must not interact with the GCP resources (Cloud Functions, Firestore Collections, etc…) of Laeticia’s “feature-2” within the GCP reco-dev project.

Sub-environment partitioning solution

The solution we have choosen to implement this partitioning is to suffix each deployed GCP resources with its branch name (e.g. “-feature-1” or “-feature-2”).

Be careful with your resource name length (GCP limit is 62 caracteres).

This way we ensure that the two sub-environments are independent.

For example:

Cloud Function will be of the form:

Name: reco-reconcile-feature-1
HTTP Url: https://europe-west1-reco-dev.cloudfunctions.net/reco-reconcile-feature-1

Collections will be of the form:

Name: configuration-feature-1

Technical implementation of sub-environments with Terraform

With Terraform , each sub-environment has its own workspace (e.g. “feature-1”) with its “.tfstate” file (e.g. “feature-1.tfstate”) stored in a bucket.

To manage the suffix in Terraform, a local variable is created from the “terraform.workspace” variable:

suffix = “-terraform.workspace”

This variable is used to concatenate each name of each GCP resource with this suffix.

Thus, each deployment of a new sub-environment results in the creation of a new workspace, a new tfstate file & GCP resources specific to that sub-environment.

Single GCP resource constraint

The main constraint of this multi-branch deployment on GCP is the management of unique resources per GCP project.

Indeed, some GCP resources need to be unique per GCP project and therefore shared by all sub-environments of it.

In our case, we have several GCP resources that need to be unique to the GCP project, like the VPC & BigQuery dataset for example.

Currently, if no exceptions are implemented in the Terraform for these unique resources, they will be deployed systematically for each sub-environment which will cause a problem at deployment time.

Single GCP resource solution

To manage this particular case, it is necessary to add a notion of “main branch” per GCP environment.

This main branch is a sub-environment which will deploy all the unique GCP resources to the project and which will always be present on the project (so never destroyed).

All other deployed sub-environments will not be considered as “main branch” so they will not deploy these unique resources but rather use the one deployed by the “main branch”.

In our GCP project these are the main branches:

reco-dev ⇒ git “develop” branch with a workspace named “develop”
reco-qa ⇒ last git tag with a workspace named “qa”
reco-stage ⇒ last git tag with a workspace named “stage”
reco-prod ⇒ last git tag with a workspace named “prod”

The workspace name of a sub-environment or main branch must be fixed so that it is always present and up to date with the latest version and other sub-environments can access their unique resources.

Technical implementation with Terraform of single resources

On the Terraform side, we need to unbundle the main branches. For this, a local variable “is_main_branch” is created to unlink the type of sub-environment to be deployed:

Thus, on each deployment, Terraform is able to determine whether the branch to be deployed is a main branch or not.

Then, on each GCP resource that’s unique to the project (such as the VPC & BigQuery dataset) , the “count” parameter should be added as follows:

Thus, the main branches will systematically deploy the unique GCP resources (here the VPC) and the other branches or sub-environments will never deploy these resources but will be able to use them.

For example, deploying a “develop” branch and then a “feature-1” branch will do the following actions :

The develop branch will deploy all resources + unique resources (here the VPC)
The feature-1 branch will deploy all resources except the unique resources

In this case, we’ll have a single VPC resource deployed in the reco-dev project.

All the resources that need this VPC (both the develop branch and the feature-1 branch) will use the same VPC resource.

Constraint

Different constraints have been identified in our choices.

The number of service accounts per GCP project is limited to 100, so it is necessary to destroy branches when they are no longer in use. Depending on your project size, it may be necessary to optimize the service accounts in your Terraform.
Adding a new unique GCP resource to a feature requires some tinkering. This is because the feature branch is not a main branch and will not deploy the unique GCP resource. However, in our case, we need it because this unique GCP resource is not on the main branch. So we need to force the “count” property to 1 in Terraform for this feature. Then we’ll have to modify it in the git merge on the develop branch.