Integrating Visual Studio Team Services with Terraform Enterprise

Ryan Hall
Ryan Hall
Sep 5, 2018 · 6 min read

I’ve been working as a solution engineer at HashiCorp for about 7 months now and a common question that comes up when I talk to customers about Terraform Enterprise (TFE) is which version control systems (VCS) we support. In reality, this conversation is pretty short-lived. I usually find that customers have multiple VCS providers and therefore integrating with TFE is fairly straightforward. The thing is, what do you do if your organisation has settled on a VCS that is not supported out of the box?

The intent of this article is to talk about some of the workflows that we can use within TFE that will allow you to continue using your VCS of choice, and yet still take advantage of all the benefits of TFE.

First things first, let’s unpack a few core concepts.

Workspaces

Terraform implements a workspace construct which serves a few purposes. In its simplest form, the workspace represents a folder in a VCS where you store your Terraform templates (main.tf, variables.tf, outputs.tf etc.). You may also see this referred to as a working directory in our documentation. Workspaces also provide a few other key features, such as:

  1. Storage — Providing collaborating teams with a centralised and versioned space to store state.
  2. Insights — Historical view of all activities (Run) that have been performed from the day infrastructure was provisioned, until the day it is decommissioned.
  3. RBAC — Security boundary that provides TFE administrators the ability to delegate access to teams of developers and share resource metadata.
  4. Variable Store — Provides a means of defining both Terraform and Environment variables, and most importantly, allows customers to store secrets in a secure manner.

All in all, workspaces and the functionality they provide, remove many of the headaches that organisations experience when managing infrastructure-as-code at scale.

Version Control

This one is simple. The VCS is where we store your application code and the associated Terraform templates that are required to deploy your infrastructure. When we integrate your VCS with TFE, we know where to go to get the templates (working directory) when we queue a Run via the Web UI or a push or pull request.

All of this activity is provided by a webhook on the VCS repository and authentication is handled via oAuth.

Runs

This is where we get to the pointy end of the spear, which is workflows. Now if you have seen any of the videos that are available on our YouTube channel you will notice that we are all about workflows. In TFE we have three core workflows:

  1. UI-driven — All provisioning operations are carried out by the user via the Web UI.
  2. CLI-driven — All operations are performed via the shell on your workstation using a local working directory which contains all templates.
  3. API-driven — All operations are performed by making calls to the TFE API. This workflow allows you to integrate custom or 3rd-party systems with TFE. These could be IT Service Management platforms like Service Now or one of the many CICD platforms available.

The Problem Space

Right, so now that we have the core concepts out the way, let’s get down to the business end of solving our problem. Enter the humble TFE API!

The solution I will outline below is something I used with a financial services institution that has a multi-cloud strategy. My customer already had a great deal of investment in VSTS, PowerShell, and ARM templates when deploying to Azure. One of the core use-cases was to show that they can continue to use VSTS for CICD but hand off to Terraform Enterprise to handle the provisioning of resources. The Proof of Concept (PoC) was initially intended to prove the workflow for Microsoft Azure but this has since evolved to include AWS, GCP, and VMWare on-premises. All of the API integration is managed via PowerShell and the execution is handled by VSTS. There are a couple of reasons why we did this, namely:

  1. PowerShell is already widely adopted and I’m all about the KISS principle (and working smart not hard). The last thing I want is to add the extra overhead of learning a new scripting language when they are already getting accustomed to HCL in Terraform.
  2. With native support for PowerShell in Windows and PowerShell Core support in most Linux distros, I can run these scripts in VSTS and just as easily refactor them (with very little effort), and have them run within another CICD pipeline (see: Work Smart Not Hard).

Prerequisites

To test this for yourself, you will need a few things:

  1. Terraform Enterprise — You will need a TFE subscription to complete this demonstration. If you don’t have one, I recommend that you request a trial.
  2. Terraform Organisation — In TFE we use the organisation construct to manage access control, auditing, billing, and authorization.
  3. Bearer Token — This is the key to the Terraform castle which allows you to connect to the TFE API. We support many different tokens but for the purposes of this demo, I will be using a user token.
  4. Visual Studio Team Services (VSTS) — Microsoft is kind enough to give away a lot of their tools for free and VSTS is no exception. All you need to do is sign-up!
  5. Microsoft Azure — Again, Microsoft hands out free credits. All it takes is some personal information and you are ready to go.
  6. Azure Service Principle — Terraform uses a service principle in Microsoft Azure Active Directory to authenticate and provide the rights required to create resources within your Azure subscription. Microsoft has written a really detailed guide on how to create service principle using the Azure CLI. Be sure to keep this information safe and please don’t put it in a public repository. I know this seems obvious, but you would be surprised how many times I hear about people getting locked out of their subscription because someone managed to get ahold of their keys.
  7. Code — This is the final piece of the puzzle and consists of the automation scripts, VSTS configuration, and most importantly Terraform templates. I have taken the liberty of uploading some demo content to my personal GitHub repository that you can import into your VSTS repository.

Demonstration Time!

So, let’s look at what this workflow looks like in action. In the following demo, I’ll show the creation of a new project in VSTS, I’ll import some code and create a build, and release pipeline with a couple of variables. Finally, I will enable continuous deployment on the pipeline so each time I commit some code it will automatically run and provision an Azure Resource Group and Virtual Network. Too easy!

Conclusion

The workflows in TFE provide customers with a lot of flexibility and the adoption is largely dependant on the operational maturity of the practitioners that are consuming the service. If you are just dipping your toe in the water, you would most likely start in the UI. As you progress and your skills grow, you’ll move into more complex integrations where you have an ITSM platform talking to a server-less function, that is talking to an automation layer, calling Terraform, that will cookie cut repeatable infrastructure at the click of a mouse. Happy Days!

HashiCorp Solutions Engineering Blog

A Community Blog by the Solutions Engineers of HashiCorp and Invited Guests

Ryan Hall

Written by

Ryan Hall

Pre-Sales Solutions Engineer at HashiCorp.

HashiCorp Solutions Engineering Blog

A Community Blog by the Solutions Engineers of HashiCorp and Invited Guests

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade