Terraform Cloud Project Bootcamp with Andrew Brown — 2.4.0 CRUD Resource

This article is part of my Terraform journey with Terraform Bootcamp by Andrew Brown and Andrew Bayko with Chris Williams and Shala.

My wild career adventure into Cloud Computing continues with Andrews, and I highly, highly recommend you to come join us if you are interested. Check out Andrews’ free youtube learning contents. You can also buy some paid courses here to support their good cause.

Agenda

Video here: 2 4 0 CRUD Resource

Issue #49 Goals

• ✅ 1) Finish the implementation of Provider Plugin by
• ✅ 2) implementing the CRUD functions.
• ✅ 3) Test the Plugin and troubleshoot.

Workflow

• ✅ 1) Update Resource() function in main.go
• ✅ 2) Create resource in main.tf
• ✅ 2–1) Troubleshooting: Failed to query available provider packages
• ✅ 3) Implement CRUD functions
• ✅ 4) Go troubleshootiong

Faithful campers follow Andrew along every step of his way! Andrew’s been craving for payday bars and this is the closest I can find where I’m based in. Our SCT (Standard Camping Time ⛺) is usually the wee morning hours for me (Canada and Korea) but whatever. I’ll burn it at the gym later🏃

1. Update Resource() function in main.go

In reality, it is the best if we can break down every resource we implement into atomic building blocks. In Andrews’ bootcamp, however, we just put everything into the single Resource() for simplicity and learning purposes.

Define the details of Resource() as below. The major updates to Resource() in this episode are:

• Define Schema, which will be used as the attributes of the resource block in HCL later in the main.tf file.

func Resource() *schema.Resource {
 log.Print("Resource:start")
 resource := &schema.Resource{
  CreateContext: resourceHouseCreate,
  ReadContext:   resourceHouseRead,
  UpdateContext: resourceHouseUpdate,
  DeleteContext: resourceHouseDelete,
  Schema: map[string]*schema.Schema{
   "name": {
    Type: schema.TypeString,
    Required: true,
    Description: "Name of home",
   },
   "description": {
    Type: schema.TypeString,
    Required: true,
    Description: "Description of home",
   },
   "domain_name": {
    Type: schema.TypeString,
    Required: true,
    Description: "Domain name of home eg. *.cloudfornt.net",
   },
   "town": {
    Type: schema.TypeString,
    Required: true,
    Description: "The town to which the home will belong to",
   },
   "content_version": {
    Type: schema.TypeInt,
    Required: true,
    Description: "The content version of home",
   },
  }
 }
 log.Print("Resource:end")
 return resource
}

Then run ./bin/build_provider to update the plugin.

2. Create resource in main.tf

Now that our Plugin’s Resource is updated, we can use the updated feature immediately:

• Create a resource block in HCL in main.tf.
• Provide values to Schema’s attributes.

resource "terratown_home" "home" {
  name        = "Nomadiachi's nomadic life"
  description = <<DESCRIPTION
    Mariachi works nomadically so popping in and checking out here and there.
    Pictures of some memorable moments, quick snapshots of days that pass by. 
    Come join my nomadinary journey? :D 
DESCRIPTION
  # domain_name = module.terrahouse_aws.cloudfront_url
  domain_name     = "3fdq3gz.cloudfront.net" // mock url
  town            = "the-nomad-pad"
  content_version = 1
}

2–1. Troubleshooting: Failed to query available provider packages

Then we run terraform init and run into the below error.

Andrew decides to keep implementing the code to see if the problem persists.

3. Implement CRUD functions

For below walkthrough, I will be using only the Create function as an example. What I explain below is the common flow of the HTTP requests using CRUD functions. Please note that all the four CRUD functions must be updated, and they are slightly different from each other as each function performs different tasks. If you run into trouble executing the code later, feel free to copy Andrew’s entire source code.

Add config

• Config contains three attributes: user_uuid, token, and endpoint.
• Config is used in the payload for HTTP request (the data that’s passed to manipulate the TerraTowns database) which requires credential data.

Make HTTP request

• In the req, err block below, see how config is used. HTTP request is sending a request to an endpoint, so we always need a URL.

Set request headers

• Headers are used to provide additional information about the message being sent or received.
• We include the config.Token data for authentication at the TerraTowns website.

func resourceHouseCreate(ctx context.Context, d *schema.ResourceData, m interface{}) diag.Diagnostics {
 var diags diag.Diagnostics

  // Add config
  config := m.(*Config) 

  // Make HTTP request
  req, err := http.NewRequest("POST", config.Endpoint + "/u/" + config.UserUuid + "/homes", bytes.NewBuffer(payloadBytes))
    if err != nil {
      return diag.FromErr(err)
    }

  // Set headers
  req.Header.Set("Authorization", "Bearer "+config.Token)
  req.Header.Set("Content-Type", "application/json")
  req.Header.Set("Accept", "application/json")

  // Get response
  client := http.Client{}
  resp, err := client.Do(req)
  if err != nil {
  return diag.FromErr(err)
  }
  defer resp.Body.Close()

 return diags
}

Payload

• We create payload which is a map of string key — value pairs.
• Payload is used only in the Create and Update functions.
• json.Marshal() function handles the conversion of Go data structures into a valid JSON format.
• Note that the payload and its marshaling happens prior to making the HTTP request. The json-formatted payload will be included in the http request (req) down the code to the TerraTowns website.

 payload := map[string]interface{}{
  "name":            d.Get("name").(string),
  "description":     d.Get("description").(string),
  "domain_name":     d.Get("domain_name").(string),
  "town":            d.Get("town").(string),
  "content_version": d.Get("content_version").(int),
 }

 payloadBytes, err := json.Marshal(payload)
 if err != nil {
  return diag.FromErr(err)
 }

Parse response JSON

• responseData is a map of interface{}, which is a type that can hold values of any type. It will hold the parsed Json data.
• Just like we converted Go into Json (marshaled) when sending HTTP request, we have to convert the response Json back for Go (parse).
• json.NewDecoder(resp.Body) will do the job.
• The json.NewDecoder(resp.Body) will .Decode the &responseData.
• if any error occurs while decoding, the error will be stored in err.
• if there is no error (nil)

StatusOK = 200 HTTP Response Code

• To check if Status is OK, we are actually checking if it’s not okay.
• If it’s not okay, the plugin will tell us the status code that it received from the TerraTowns server.

// parse response JSON
 var responseData map[string]interface{}
 if err := json.NewDecoder(resp.Body).Decode(&responseData); err != nil {
  return diag.FromErr(err)
 }

 // StatusOK = 200 HTTP Response Code
 if resp.StatusCode != http.StatusOK {
  return diag.FromErr(fmt.Errorf("failed to create home resource, status_code: %d, status: %s, body: %s", resp.StatusCode, resp.Status, responseData))
 }

The CRUD implementation is completed at 45:45. The complete code is found in Andrew’s main.tf.

4. Go troubleshootiong

Now that the CRUD functions are complete, we want to test our plugin package. Run ./bin/build_provider.

We run into the following errors:

• Some if err statements were missing nil.
• Import libraries if missing: “encoding/json”, “net/http”
• client := http.Client{}: Client plus curly, not parenthesis.
• For undefined: responseData: compare your code against Andrew’s. responseData is used in Create and Read only.
• Add library “bytes”.
• Correct diag.FrontErr to diag.FromErr
• For undefined: responseData, add interface{} with curlies:

var responseData map[string]interface{}

4. Test out our custom provider

Run terraform init to test if the provider is compiled correctly. And we are still stuck with the same error:

• Failed to query available provider packages

In our ./terraform-provider-terratowns/, check main.go file and correct terratown_home(singluar) to terratowns_home(plural).

func Provider() *schema.Provider {
 var p *schema.Provider
 p = &schema.Provider{
  ResourcesMap: map[string]*schema.Resource{
   "terratowns_home": Resource(), // terratown should be plural 
  },

Build the plugin again, then run terraform plan to see if: terraform is creating any resource! It works and you can hear Andrew’s excitement (56:14)! This is why we learn to code!!! 😃

“It is showing us… our resource!!! So we just made our custom resource. and it’s showing what is going to change.”

According to Andrew’s words,

I remember that our mock server is not real, so it will not work exactly correctly. But we can play a little bit around before trying it on the real TerraTowns.

Now let’s run terraform apply.

Troubleshooting: Plugin did not respond.

It wasn’t going to be all too easy, was it? It’s not all sunshine and roses!

In resourceHouseCreate.payload, check if content_version type is int or string.

 payload := map[string]interface{}{
  "name":            d.Get("name").(string),
  "description":     d.Get("description").(string),
  "domain_name":     d.Get("domain_name").(string),
  "town":            d.Get("town").(string),
  "content_version": d.Get("content_version").(int),
 }

Troubleshooting: failed to create home resource, status_code: 401 Unauthorized

I kept running into the above error. At this stage, we are still using the mock values Andrew used in the videos for domain_name, user_uuid and token.

• Check the value of domain_name which is a mock cloudfront.net url.
• Check the List of files where user_uuid & token are used:
• ./bin/terratowns/create
• ./bin/terratowns/delete
• ./bin/terratowns/read
• ./bin/terratowns/update
• ./terratowns_mock_server/server.rb
• ./main.tf

Troubleshooting: json: Unmarshal(non-pointer map[string]interface {})

The error message json: Unmarshal(non-pointer map[string]interface {}) indicates that there’s an issue with unmarshaling JSON data into a non-pointer map[string]interface{} type in your Terraform configuration.

So I scan through the code and I see the yellow underline below Decode(). I hover over and I get the tip the call of Decode passes non-pointer. It turns out, I was missing the ampersand sign (&) which is a pointer in Golang.

The Decode function expects a “pointed (&)” argument so make sure to attach an ampersand before the argument.

💡Quick tip: difference between pointer and non-pointer arguments

• pointer argument: it points to the actual value. So if the value is updated, your functions are working with the updated value.
• non-pointer argument: it is a copy of the argument. The copy’s value will stay the same even after the argument’s current value is updated.

decode function

So I add an ampersand as a pointer (&) as below and run terraform apply. It works a charm 😃!

// resourceHouseCreate()
// parse response JSON
 var responseData map[string]interface{}
 if err := json.NewDecoder(resp.Body).Decode(&responseData); err != nil {
  return diag.FromErr(err)
 }

This is how I spotted the difference. WinMerge is amazing!

Now that the complete Provider Plugin is working, we are good to define terratown_homes and deploy them to TerraTowns.

terraform apply

Our plugin allows us to provision terratowns home(s).

📑 Notes

Go language

• json.Marshal() converts Go into json-formatted bytes.
• Schema in Go is used to define inputs (attributes of code blocks such as provider or resource) in Terraform provider.

Go code flow

• payload exists only in Create and Update.
• responseData is used in Create and Read only.

Resource

Bootcamp

• Video: 2 4 0 CRUD Resource
• My feature branch: 2 4 0
• My complete Learning Journal

Golang

• func (*Decoder) Decode