Automate Infrastructure as Code with GCP Deployment Manager

For any application that we develop, we need some infrastructure which we use to structure / deploy / host / execute the application. Also, with increased scalability demands of the applications these days, infrastructure is also expected to be scalable, reproducible, easy to review. Almost all public cloud providers such as AWS, Azure, or Google Cloud provide their own ‘infrastructure automation’’ tools as part of their offering. Using these tools, you can write infrastructure as code and consider provisioning infrastructure as part of ‘software development’. Because it is software, it should by also version-controlled, tested, and reviewed.

AWS has ‘Amazon Cloud Formation’, Azure has ‘Azure Resource Manager’, and Google Cloud has “Deployment Manager”. They all provide a way to define your cloud infrastructure resources (using a declarative language) as code which can be put together with application codebase. Also, you can use templates, which allows you to avoid code duplication, and makes it possible to test the same infrastructure code in various stages of deployment.

In this article, we will focus on Google Cloud Deployment Manager (called DM henceforth) and how you can use DM to automate the configuration of all your GCP resources. As a practical demonstration, we will deploy:

1. a VM Machine
2. a BigQuery dataset with a sample table
3. a pub/sub topic and subscription
4. a Google cloud storage bucket

How Deployment Manager works?

Before we go practical about creating various GCP resources using Deployment Manager, let’s start with a very quick summary of how it works.

It has the concept of a deployment, which is a collection of GCP resources that form a logical unit and that are deployed together. Using these deployments, we can provision most of the resources on GCP. Deployment manager supports a wide array of GCP resources. Complete list of allowed resource types is here. Definition of what resources are needed is written in a deployment configuration YAML file.

A Quick Practical

I assume:

1. you already have a GCP project and with billing enabled. If not, please follow the instructions defined here.
2. you already have Google cloud SDK installed (on your local machine). If not installed, please follow the instructions defined here.
3. you already have git installed on your local machine. If not, please follow the instructions defined here.

Now, clone github repo that I prepared for all code samples to be discussed in this article:

$ git clone https://github.com/pankajwithgit/googlecloud.git

Create a Compute Engine VM

We will create a GCE virtual machine using DM deployment. A very simple definition looks as follows:

Here are few details about the code above:

1. config.yaml is a configuration file which defines what all resources that are needed with reference to jinja template file(s).
2. vm.jinja is a template file to define all the details related to virtual machine. We define resource name, type and couple of properties. For a VM, properties can be zone, machineType, serviceAccounts, disks etc.

Here is how we execute this code (assuming you cloned my repo):

$ cd deploymentmanager/compute_engine_vm
$ gcloud deployment-manager deployments create test-vm --config config.yaml

This will create a new deployment in your GCP project. You can go to ‘Deployment Manager’ section in GCP project to see the deployment (with name as test-vm) we just created. You can also go to Compute Engine to see newly created virtual machine.

Create a BigQuery Dataset and Table

Now, let’s move ahead and create a BigQuery Dataset and table. Here is how the code would look like:

Here are few details about the code above:

1. config.yaml is a configuration file which defines what all resources that are needed with reference to jinja template file(s).
2. bigquery.jinja is a template file to define all the details related to BigQuery dataset and table.

Here is how we execute this code:

$ cd deploymentmanager/big_query
$ gcloud deployment-manager deployments create test-bq --config config.yaml

Now, if you can go to BigQuery, you can see the dataset and a employee table created.

Create a Pub/Sub Topic and it’s Subscription

Pub/Sub Topic and it’s subscription is next we will create. Here is how the code would look like:

Here are few details about the code above:

1. config.yaml is a configuration file which defines what all resources that are needed with reference to jinja template file(s).
2. pubsub.jinja is a template file to define all the details related to pub/sub Topic and it’s subscription. PS: we have used $(ref…) to refer to topic we actually are creating with same script. This will first create topic and then it’s subscription and there is no need to specify the order.

Here is how we execute this code:

$ cd deploymentmanager/pub_sub
$ gcloud deployment-manager deployments create test-pub-sub --config config.yaml

Now, if you go to Pub/Sub service on GCP, you will see 2 resources i.e. Topic and it’s subscription created.

Create a Cloud Storage Bucket

Now, we will create a GCS bucket using DM code. A very simple definition looks as follows:

Here are few details about the code above:

1. config.yaml is a configuration file which defines what all resources that are needed with reference to jinja template file(s).
2. gcs.jinja is a template file to define all the details related to GCS bucket such as name, location and storage class of the bucket.

Here is how we execute this code:

$ cd deploymentmanager/gcs
$ gcloud deployment-manager deployments create test-gcs --config config.yaml

Clean up

Now that you have created 4 deployments in your GCP project and you may want to delete those. Use following command to delete deployments one by one.

$ gcloud deployment-manager deployments delete <deployment-name>

Just replace <deployment-name> with name of your deployments. OR you can also delete all 4 deployments together using web console.

References

Here are a bunch of references for detailed implementation based on Deployment Manager:

1. Official documentation
2. Official samples
3. Best practices for using Deployment Manager

Wrapping up

Hope you now have a decent understanding of what of Google Cloud Deployment Manager and how to use it. Now, don’t forget to clean up deployments after you have finished experimenting in order to save on your GCP credits :-)