Managing Cloud SQL resources with Terraform
Terraform have had a special place in my life for a long time, the album by the eminent band Shellac that is. The infrastructure as code software tool created by HashiCorp, I’ve been aware of but never really used.
So in this article we will look at my first experience with Terraform and how we can use it to manage Cloud SQL resources. In order to follow along I’ll assume you have a Google Cloud Platform account. If you don’t have a GCP account, create one now. Furthermore I’ll assume you have Terraform installed on your machine, if not you can find instructions here.
Terraform will access your GCP account by using a service account key. Create one now in the console. When creating the key, use the following settings:
- Select your project.
- Under “Service account”, select “New service account”.
- Give it any name you like.
- For the Role, choose “Project -> Editor”.
- Leave the “Key Type” as JSON.
- Click “Create” to create the key and save the key file to your system.
Note: Treat the key file like any other secret credentials, e.g. don’t check it into source control.
Goal
So the goal for this article is to:
- Create a Cloud SQL instance with a randomized name
- Use labels to indicate that this is a development environment
- Set the maintenance window
- Create a database and a database user
- Parameterize our configurations using input variables, so we don’t have to hardcode in everything like project-id, region, etc.
- Use output variables as a way to organize data to be easily queried and shown back to us.
Configuration
So let’s start by creating a directory where we can store all configuration files. By default Terraform will load all files ending in .tf or .tfvars. In our directory let’s create a main.tf file.
provider "google" {
version = "3.10.0"
credentials = file("[NAME].json") project = "[PROJECT_ID]"
region = "us-east1"
zone = "us-east1-d"
}resource "random_id" "db_name_suffix" {
byte_length = 4
}resource "google_sql_database_instance" "postgres" {
name = "postgres-instance-${random_id.db_name_suffix.hex}"
database_version = "POSTGRES_11" settings{
tier = "db-f1-micro"
}
}
Note: We create the random id in order to not run into name resolution conflicts since there’s a lease time on names on instances.
The provider block is used to configure the named provider, in our case google. A provider is responsible for creating and managing resources. Multiple provider blocks can exist if a Terraform configuration manages resources from different providers.
The resource block defines a resource that exists within the infrastructure. A resource might be a physical component such as a server, or it can be a logical resource such as a Heroku application.
The GCP provider documents supported resources, including google_sql_database_instance.
Initialization
The first command to run for a new configuration — or after checking out an existing configuration from version control — is terraform init, which initializes various local settings and data that will be used by subsequent commands.
Initialize your new Terraform configuration by running the terraform init command in the same directory as your main.tf file.
$ terraform apply Initializing the backend...Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "google" (hashicorp/google) 3.5.0...Terraform has been successfully initialized!You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
The Google Cloud Platform provider plugin has been downloaded and installed in a subdirectory of the current working directory.
Creating resources
$ terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ createTerraform will perform the following actions:# google_sql_database_instance.postgres will be created
+ resource "google_sql_database_instance" "postgres" {
+ connection_name = (known after apply)
+ database_version = "POSTGRES_11"
+ first_ip_address = (known after apply)
+ id = (known after apply)
+ ip_address = (known after apply)
+ master_instance_name = (known after apply)
+ name = (known after apply)
+ private_ip_address = (known after apply)
+ project = (known after apply)
+ public_ip_address = (known after apply)
+ region = (known after apply)
+ self_link = (known after apply)
+ server_ca_cert = (known after apply)
+ service_account_email_address = (known after apply)(.......)# random_id.db_name_suffix will be created
+ resource "random_id" "db_name_suffix" {
+ b64 = (known after apply)
+ b64_std = (known after apply)
+ b64_url = (known after apply)
+ byte_length = 4
+ dec = (known after apply)
+ hex = (known after apply)
+ id = (known after apply)
}Plan: 2 to add, 0 to change, 0 to destroy.Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.Enter a value: yes
Creating the resource can take a few minutes. After Terraform is done we can go to the console to see the instance that has been created or we can use gcloud.
$ gcloud sql instances list
NAME DATABASE_VERSION LOCATION TIER (...)
postgres-instance-5e2a9971 POSTGRES_11 us-east1-d db-f1(...)If you look in your current working directory, you’ll see that Terraform also wrote some data into the terraform.tfstate file. This state file is extremely important; it keeps track of Terraform’s understanding of the resources it created. We recommended that you use source control for the configuration files, but the state file should not be stored in source control. You can also setup Terraform Cloud to store and share the state with your teams.
We can also inspect the current state by running terraform show.
Adding resources
Lets start by indicating that this is a development environment by setting a label. user_labels is a key/value pair that the required settings block supports.
So our updated main.tf would then look like this
provider "google" {
version = "3.10.0"
credentials = file("[NAME].json")project = "[PROJECT_ID]"
region = "us-east1"
zone = "us-east1-d"
}resource "random_id" "db_name_suffix" {
byte_length = 4
}resource "google_sql_database_instance" "postgres" {
name = "postgres-instance-${random_id.db_name_suffix.hex}"
database_version = "POSTGRES_11"settings{
tier = "db-f1-micro"
user_labels = {
"environment" = "development"
}
}
}
After updating our main.tf file we can run terraform apply again. If you go to the console now you should see something like this:
The optional settings.backup_configuration subblock supports enabling automatic backups and let’s us set a start time. So lets add that subblock to our main.tf, which gives us:
provider "google" {
version = "3.10.0"
credentials = file("[NAME].json")project = "[PROJECT_ID]"
region = "us-east1"
zone = "us-east1-d"
}resource "random_id" "db_name_suffix" {
byte_length = 4
}resource "google_sql_database_instance" "postgres" {
name = "postgres-instance-${random_id.db_name_suffix.hex}"
database_version = "POSTGRES_11"settings{
tier = "db-f1-micro"
user_labels = {
"environment" = "development"
}
backup_configuration {
enabled = true
start_time = "04:30"
}
}
}
Running terraform apply again and then going to back to console you should see something like this:
Let’s add two more resources to our instance. One database and one database user. That would leave us with the following main.tf
provider "google" {
version = "3.10.0"
credentials = file("[NAME].json")
project = "[PROJECT_ID]"
region = "us-east1"
zone = "us-east1-d"
}resource "random_id" "db_name_suffix" {
byte_length = 4
}resource "google_sql_database_instance" "postgres" {
name = "postgres-instance-${random_id.db_name_suffix.hex}"
database_version = "POSTGRES_11"
settings{
tier = "db-f1-micro"
user_labels = {
"environment" = "development"
}
backup_configuration {
enabled = true
start_time = "04:30"
}
}
}resource "google_sql_user" "users" {
name = "janedoe"
instance = google_sql_database_instance.postgres.name
password = "changeme"
}resource "google_sql_database" "database" {
name = "my-database"
instance = google_sql_database_instance.postgres.name
}
Running terraform apply, going back to our instance in the console and click Databases should show you something like this:
So far we have a configuration that creates a Cloud SQL instance of type PostgreSQL 11 that has a randomized name. Furthermore we have labels attached to the instance that indicates that it is a development environment we also automatic backups enabled and it creates a database and a database user.
Input variables
Right now we have a lot of hardcoded values in our configuration that won’t be a problem given the scale we are operating at in this article but for larger configurations it would. Let’s start by converting a few of the hardcoded values into variables. Create another file called variables.tf with the following contents:
variable "project" {
}variable "credentials_file" {
}variable "region" {
default = "us-east1"
}variable "zone" {
default = "us-east1-d"
}
Next up lets use these variables in our configuration. Update the provider part of main.tf
provider "google" {
version = "3.10.0"
credentials = file(var.credentials_file)project = var.project
region = var.region
zone = var.zone
}
There are several ways to assign variables, depending on your needs. In this article we will assign values from a file.
For security reasons, we recommend never saving usernames and passwords to version control. Your terraform configuration will probably need these secret values, though. One solution is to create a local secret variables file and use -var-file to load it.
So lets create a secret.tfvars file with the following content.
project = "[PROJECT_ID]"
credentials_file = "[NAME].json"Note: Be sure to replace [PROJECT_ID] with your project’s ID, and [NAME] with the name of your credentials file.
Output variables
Outputs are a way to tell Terraform what data is important. This data is outputted when apply is called, and can be queried using the terraform output command.
As we have configured our instance right now, the only way to connect to it would be to use the Cloud SQL Proxy. One prerequisite in order to use the proxy is that we have the instance connection name. So that seems to be a good candidate for an output variable.
Create a file called outputs.tf with the following contents:
output "connection_name" {
value = google_sql_database_instance.postgres.connection_name
}Now we can run terraform refresh -var-file="secret.tfvars", this will refresh our state by comparing it to your cloud infrastructure. In the process, it will also pick up the new output. You should see output like this:
random_id.db_name_suffix: Refreshing state... [id=XiqZcQ]
google_sql_database_instance.postgres: Refreshing state... [id=postgres-instance-5e2a9971]
google_sql_user.users: Refreshing state... [id=bergqvist//postgres-instance-5e2a9971]
google_sql_database.database: Refreshing state... [id=projects/bergqvist-sandbox/instances/postgres-instance-5e2a9971/databases/my-database]Outputs:connection_name = [PROJECT_ID]:[REGION]:[INSTANCE_NAME]
Before wrapping up let’s add our preferred maintenance window to the configuration. Although this is a development instance it would still be nice that any potential security patches and so forth would be applied when we are least likely to work on it.
That would give us this final main.tf:
provider "google" {
version = "3.10.0"
credentials = file(var.credentials_file) project = var.project
region = var.region
zone = var.zone
}resource "random_id" "db_name_suffix" {
byte_length = 4
}resource "google_sql_database_instance" "postgres" {
name = "postgres-instance-${random_id.db_name_suffix.hex}"
database_version = "POSTGRES_11" settings{
tier = "db-f1-micro"
user_labels = {
"environment" = "development"
}
maintenance_window {
day = "1"
hour = "4"
}
backup_configuration {
enabled = true
start_time = "04:30"
}
}
}resource "google_sql_user" "users" {
name = "janedoe"
instance = google_sql_database_instance.postgres.name
password = "changeme"
}resource "google_sql_database" "database" {
name = "my-database"
instance = google_sql_database_instance.postgres.name
}
After running terraform apply -var-file="secret.tfvars", we could go back into the console and see that we now have a preferred maintenance window like so:
Summary
So these were my first baby steps using Terraform to manage my Cloud SQL resources. I found it really straight forward and I can totally see why it has gained so much traction. I hope you found it enjoyable. If there is anything that is unclear or if you have feedback feel free to contact me.
All the quoted parts in this article comes from the learn Terraform for GCP documentation.
There is also a Github repo containing all the code
