Calling External APIs in Terraform Cloud Sentinel Policies

Roger Berlind
Dec 3, 2019 · 6 min read
HashiCorp Terraform
HashiCorp Terraform

Introduction

HashiCorp recently added two important new features, a new HTTP import and Parameters, to the 0.13.0 release of its policy-as-code solution, Sentinel. In this blog post, we discuss some example Sentinel policies that use these new features in Terraform Cloud.

The HTTP import allows Sentinel policies to retrieve data from external API endpoints that return JSON documents.

Parameters were primarily added to enable policy authors to define API credentials securely outside of the policies themselves since these are often stored in VCS repositories. However, parameters can also be used for other purposes, making Sentinel policies more flexible.

At the same time, Terraform Cloud now allows parameters to be added to Policy Sets and allows them to be marked as sensitive. In other words, Terraform Cloud allows for the secure definition of the parameters that Sentinel policies can now accept.

Some customers want their Sentinel policies to call external APIs in order to check that change requests have been approved in their internal systems. Our first example below simulates that. Our second example below calls the Terraform Registry API to make sure that the most recent versions of Terraform modules are being used. Other possibilities include calling cloud APIs to fetch information about resources and services and calling Vault’s HTTP API to retrieve secrets.

Example Policies

You’ll find two policies that use the HTTP import in the hashicorp/terraform-guides repository here.

The first of these, check-external-http-api.sentinel, is quite simple and rather silly; it simply uses the HTTP import to call the public API, https://yesno.wtf/api, that randomly returns “yes” or “no” (but sometimes “maybe”). This API actually returns a JSON document that contains the answer, a flag called force, and a URL of an image. A typical response looks like this:

{
"answer":"yes",
"forced":false,
"image":"<URL of image>"
}

Here is the function in the policy that uses the HTTP import:

check_external_approval_system = func() {
req = http.request("https://yesno.wtf/api")
res = json.unmarshal(http.get(req).body)
answer = res.answer

print("answer:", answer)
case answer {
# https://yesno.wtf/api returns "maybe" every 10,000th time
when "yes", "maybe":
return true
when "no":
return false
else:
return false
}
}

We have used bold-face for that part that uses the HTTP import and its data types. Since we only care about the answer, we only extract it from the response with answer = res.answer. We then use the Sentinel case statement that was added in Sentinel 0.12.0 to map the answer to a boolean that the function returns. We return true if the answer is yes or maybe and false if it is no. That’s about as simple as it gets!

While the check-external-http-api.sentinel policy does not use any policy parameters, the second policy, use-latest-module-versions.sentinel, does. This policy calls the Terraform Registry List Modules API of a Terraform Cloud (TFC) or Terraform Enterprise (TFE) server to determine the most recent version of each Terraform module in the Private Module Registry (PMR) of an organization on that server or in the public Terraform Registry. It then validates that all modules used in the current TFC or TFE run that come from the specified registry use the most recent versions. The versions of modules from other sources are not validated.

In case you have never used a module from a Private Module Registry, here is an example of Terraform code that does that:

module "network" {
source = "app.terraform.io/Cloud-Operations/network/azurerm"
version = "1.1.1"
location = "East US"
resource_group_name = "roger-berlind-sentinel-demo"
allow_ssh_traffic = true
}

Note that app.terraform.io is the multi-tenant Terraform Cloud environment run by HashiCorp. Cloud-Operationsis the organization, network is the module name, and azurerm is the main provider for the module.

Here are the parameters the policy uses:

  • public_registry indicates whether the public Terraform registry is being used. This is false by default, but could be set to true.
  • address gives the address of the Terraform Cloud or Terraform Enterprise server. It defaults to app.terraform.io. You must specify a value for this if using a Terraform Enterprise server.
  • organization gives the name of an organization on the Terraform Cloud or Terraform Enterprise server specified by address. You must always specify a valid organization.
  • token gives a valid TFC API token which can be a user, team, or organization token. See the API Tokens document for more information.

These parameters are declared within the policy as follows:

# A boolean indicating whether to use the public Terraform registry
param public_registry default false
# The address of the Terraform Cloud or Terraform Enterprise server
param address default "app.terraform.io"
# The name of the Terraform Cloud or Terraform Enterprise organization
param organization
# A valid Terraform Cloud or Terraform Enterprise API token
param token

The actual parameter declarations are in bold-face and are accompanied by comments. These are actually used by the Sentinel CLI as parameter descriptions that it shows to users who run the sentinel apply command without providing values for parameters missing default values. In this policy, public_registry and address have default values but organization and token do not.

Once a parameter has been declared in a policy, it can be used like a variable. This includes passing them into functions as is done in this function from the policy that actually uses the HTTP import to query the registry API:

retrieve_latest_module_versions =
func(public_registry, address, organization, token) {
# Call the TFC Modules API and extract the response
if public_registry {
# We are limiting the request to 20 verified modules
req = http.request("https://" + address + "/v1/modules/" +
organization + "?limit=20&verified=true")

} else {
req = http.request("https://" + address +
"/api/registry/v1/modules/" +
organization)
req = req.with_header("Authorization", "Bearer " + token)

}
res = json.unmarshal(http.get(req).body)
modules = {} # Extract names, providers, and latest versions from all modules
for res.modules as m {
index = m.namespace + "/" + m.name + "/" + m.provider
modules[index] = m.version
}
return modules
}

Again, we’ve used bold-face above to highlight the use of parameters and the HTTP import.

We don’t analyze the rest of the policy in this blog post, but you can see all of it by following the link we gave above.

Using the Policies with the Sentinel CLI

The repository that contains the two sample policies described above also contains Sentinel mock files and test cases so that the Sentinel CLI can be used to test them. Directions for doing this are in this README.md file. We recommend that you clone or fork the repository and follow those directions to test the policies with the Sentinel CLI. (Note that you do not need a Terraform Cloud account or Terraform Enterprise server to use the CLI.)

Using the Policies in a Terraform Cloud Organization

To use the policies in a Terraform Cloud or Terraform Enterprise organization, you need to add them to a policy set and then register that policy set with the organization. You will need access to an organization that has some modules in its Private Module Registry (PMR). You can follow these steps which are also given in the README.md mentioned in the last section.

  • Fork this repository which is a greatly reduced subset of the hashicorp/terraform-guides repository mentioned above. It contains copies of the two policies and a Policy Set Configuration File, sentinel.hcl, that lists the policies and sets their Enforcement Levels.
  • Register a new policy set with an organization on your Terraform Cloud or Terraform Enterprise server, following the instructions in this document. This would typically be done with the Terraform Cloud UI.
  • Edit the registered policy set to specify values for the organization and token parameters making sure you pick an organization that actually has some modules in its PMR and that the token you specify is a valid API token with permission in that organization. Normally, the organization with the modules and the policy set would be the same. Note that you cannot specify parameters until after creating the policy set. Parameters are added at the bottom of the Policy Set screen.
  • Be sure to mark your tokenparameter as sensitive so that nobody else can see it in the Terraform Cloud UI.
  • If using a Terraform Enterprise server, also specify a value for the addressparameter, using a value like “tfe.example.com”.
  • Save the policy set.
  • Add a workspace to the policy set that uses Terraform code that references modules in the PMR in the organization you specified.
  • Queue a plan against that workspace in the Terraform Cloud UI.

Running the plan should trigger a check against your policy set. If your Terraform code uses the most recent versions of all modules in the PMR, then the policy will pass. If your code uses any older versions of modules in the PMR, the policy will fail.

Conclusion

In this blog post, we have illustrated how Sentinel’s new HTTP import and parameters can be used in Terraform Cloud Sentinel policies. We hope you found it interesting. We expect that many of our customers will use the HTTP import in their TFC and TFE Sentinel policies. Thanks for reading!

HashiCorp Solutions Engineering Blog

A Community Blog by the Solutions Engineers of HashiCorp and Invited Guests

Roger Berlind

Written by

Roger is a Sr. Solutions Engineer at HashiCorp with over 20 years of experience explaining complex technologies like cloud, containers, and APM to customers.

HashiCorp Solutions Engineering Blog

A Community Blog by the Solutions Engineers of HashiCorp and Invited Guests

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade