Which one is Terraform?

Terraform Enterprise and Friends: Using Notifications and API Calls to Integrate with External Tools

Overview

Terraform Enterprise offers powerful features that address the challenges of deploying infrastructure within teams in an organization. Examples include Sentinel for creating deployment guardrails, user permission controls per workspace, centralized variable storage, and several more.

In this blog post, we will talk about two of those features — Notifications and API endpoints. We will show how Terraform can be integrated with an external system, in this case ServiceNow, that will receive status change notifications associated with a Terraform workspace, and that will issue commands to Terraform using the API endpoints.

This diagram summarizes the workflow to be described in this blog:

Disclaimer

Not production!

The code in this blog is not meant for production use, the goal is to demonstrate how external tools can be integrated with Terraform. If you decide to use this workflow in production, please review ServiceNow’s security options for REST API scripts and best practices for limiting the scope of scripted events.

Enterprise features

Please note that Notifications and API Endpoints are not available in Terraform Open Source. In order to follow these examples, you will need to have a Terraform Enterprise SaaS account or access to Terraform Enterprise Private Install. For more information please visit the Terraform Enterprise site.

Background

Let’s assume we have company Business Co, which uses Terraform to deploy resources to Azure and AWS, and where Sarah is a member of the System Administrators team and Jason is a member of the Development team. Sarah wants to give Jason freedom to deploy resources in their “Dev” environment in Azure, however, she wants to use an approval workflow leveraging ServiceNow to approve deployments to the “Prod” environment.

Note that these approvals can all be done through Terraform web interface or API calls, however, a company might prefer to use external tools such as ServiceNow to comply with internal requirements or to follow existing workflows.

In the next sections, we will describe how to set up this workflow: first creating a Terraform API token that will be used by ServiceNow, then working on the ServiceNow side, and finally going back to Terraform Enterprise to configure Notifications.

Terraform

Create API Token

Log in to Terraform Enterprise, click on your user icon and on “User Settings”, then click on “Tokens”. Enter a name and generate the token — make sure you save the output somewhere secure.

ServiceNow

Create a Dev account

1. Go to https://developer.servicenow.com and signup
2. Create an instance using “Madrid” image
3. Your INSTANCE_ID can be found in the url or in your developer home:

Log In to Dashboard

1. Click on the instance link
2. Enter your admin username and password (set when you created the instance)
3. You will be redirected to the home dashboard. You can easily navigate using the search bar “Filter navigator” on the top left.

Scripted Rest API

Now that you have the TFE token and the ServiceNow instance, let’s configure the ServiceNow endpoint that will be listening for Terraform Notifications. This is what will be listening for TFE notifications.

1. On the “Filter Navigator”, search for “rest api”, and go to “Scripted Web Services > Scripted REST APIs”.
2. Click “New”, enter “Name” and “API ID”. These can be any value. Hit “Submit”.
3. You should be back to the previous screen. Type the name you gave to your script in the search filter, press enter, and click on the name to open it.
4. Once opened, scroll down and on Resources click “New” at the bottom of the screen.

5. Set HTTP Method to “POST”, and enter a “Name”

6. Paste the following script:

(function process(/*RESTAPIRequest*/ request, /*RESTAPIResponse*/ response) {
    var data = request.body.data;

	// Will only be triggered if it is a pending Terraform apply
    // Sample TFE payload: https://www.terraform.io/docs/enterprise/api/notification-configurations.html#sample-payload: 
    if (data.notifications[0].trigger == 'run:needs_attention') {
        var run_url = data.run_url.toString();
	    var run_id = data.run_id.toString();
	    var organization_name = data.organization_name.toString();
	    var workspace_name = data.workspace_name.toString();
	    var run_created_by = data.run_created_by.toString();
	    var message = data.notifications[0].message.toString();
	
        var grI = new GlideRecord('sys_user');

        gs.log(gs.getUserID());

        grI.addQuery('sys_id',gs.getUserID());

        grI.queryNoDomain();

        if(grI.next()){
            // Creates a new incident
            var gr = new GlideRecord('incident');
            gr.initialize();
            gr.setValue('category','TFE run');
            gr.setValue('subcategory','Confirm apply');
            gr.setValue('short_description','Confirm apply for ' + organization_name + '/' + workspace_name );

            // This will store the json object from Terraform in the description field
            var parser = new JSON(); 
            var str = parser.encode(data);
            gr.setValue('description', str );

            // Feel free to change per organization's needs
            gr.setValue('comments','Created by ' + run_created_by);
            gr.setValue('urgency','3');
            gr.setValue('impact','3');
            gr.setValue('caller_id','n/a');
            gr.setValue('contact_type','WebService');
            gr.setValue('company',grI.getValue('company'));
            gr.setValue('location',grI.getValue('location'));
            gr.setValue('caller_id',gs.getUserID());
            gr.sys_domain = gs.getUser().getDomainID();
            gr.insert();

            var number = gr.getValue('number');
            var state = gr.state;

            return {
                "incident_number":number,
                "state":state,
                "domain":gr.getDisplayValue('sys_domain')
            };
        }
	} else {
		gs.info("Received notification from Terraform, but it wasn't a pending apply");
		gs.info("Terraform request: " + request.body.data.toString());
	}
})(request, response);

7. Make sure you uncheck “Requires authentication” box (once again, this is not production-level code)

8. Hit “Submit”. This will create a listener in ServiceNow, listening for a json object received by this API endpoint. It will then create an incident and populate the fields based on the data received. Note that there is a conditional checking that the data has “run:needs_attention". This is the Terraform state of a successful Sentinel check, pending an apply. Additional Terraform states documented here.

Business Rules

Now that we have created the listener for Terraform Notification calls, let’s create a ServiceNow event that will call Terraform API once there is interaction with the incident.

1. Search for “Business rules” on the top left Filter Navigator and click on “System Definitions > Business Rules” (note that this is under “System Definitions”, not “Metrics”)
2. Search for “incident” on the table search bar
3. Click on “incident events”. These are the commands that will be executed whenever there is a change to incidents

4. Click on the link in the notification at the top of the screen to edit the script:

5. Click on the “Advanced” tab, and add the following to the script on line one:

Note: Make sure you update “YOUR_TFE_TOKEN_HERE” with the token created in Terraform Enterprise in the first step.

if (current.category == 'TFE run' && current.resolved_at != '') {
    gs.info("Preparing to send TFE API request");
	var requestBody;
    var responseBody;
    var status;
    try {
        // On the Scripted Rest API that receives TFE notification, we put the json data on the incident's description
        // In this block we will reverse that to a json object
        var parser = new JSON();
        var str = current.description;
        var obj = parser.decode(str);
        
        // Creates message
        var r = new sn_ws.RESTMessageV2();
		r.setEndpoint("https://app.terraform.io/api/v2/runs/" + obj.run_id + "/actions/apply");
		r.setHttpMethod("POST");
        // Make sure you update the TFE token
		r.setRequestHeader("Authorization", "Bearer "+ "YOUR_TFE_TOKEN_HERE");
		r.setRequestHeader('Content-Type', 'application/vnd.api+json');

        var response =	r.execute();
        responseBody = response.haveError() ? response.getErrorMessage() : response.getBody();
        status = response.getStatusCode();
    } catch(ex) {
        responseBody = ex.getMessage();
        status = '500';
    } finally {
        gs.info("API call to TFE was successful");
        requestBody = r ? r.getRequestBody():null;
    }
    gs.info("Request Body: " + requestBody);
    gs.info("Response: " + responseBody);
    gs.info("HTTP Status: " + status);
}

6. Press “Submit”. This will create a new version of the script, so you can always rollback to an older version. The above script is set to be triggered when an incident is “closed”, it will send a call to Terraform API to trigger a pending apply.

Terraform

Now let’s go back to Terraform and configure Notifications

1. Open desired workspace, click on “Settings > Notifications”

2. Click “create notification”, and make sure “Webhook” is selected

3. Enter any name. For the “Webhook URL”, enter the values generated when you created the Rest Script. It should look something like:

https://YOUR_INSTANCE_ID.service-now.com/api/YOUR_API_ID/YOUR_ENDPOINT_NAME

4. You can leave “token” blank. To read more on this functionality follow this link.

5. Click on “Only certain events”, uncheck all except “Needs attention”. This means that Terraform Enterprise will only issue a notification from this workspace after a successful Sentinel check and before an apply.

6. Make sure you enable the Terraform notification

Validation

1. Trigger a TFE run in the selected workspace
2. On ServiceNow, go to Service Desk > Incidents and see the incident created
3. Enter “Resolution notes” and “Resolution Code”, and press “Resolve”
4. Go to “System Logs > System Log > All” and validate “API call to TFE was successful” message

“Terraform is the baa-st!”

Next Steps

The above example was simple and meant to show the Terraform Enterprise Notification and API Endpoints functionality. Both are very flexible; here are a few ideas of how to expand this code:

• Create a different ServiceNow incident category for failed Sentinel checks
• Create different ServiceNow event trigger to allow denying an apply
• Use ServiceNow “System Web Services > Outbound > REST Message” to centralize location of Terraform Enterprise token
• Use ServiceNow “System Policy > Events > Script Actions” for more fine-grained control of the REST API script

Demo

You can also watch the supplemental demo to this blog. I will be hosting our HashiCorp Solutions Engineering Hangout on April 16. Register here to watch it live.