Developing a public Terraform provider — Part 1
What happens after you create a Terraform provider and release it into the wild?
What do users need to be able to use the provider, will it work in their environments? How do you develop it of that it’s robust and handles unforeseen failures or configuration scenarios? This was the situation I found myself in when I was recently extending the functionality of the IBM Cloud Terraform provider .
Hashicorp provide a good introduction to writing a custom provider. There are also many articles on the web on how to create a basic provider, but I found little guidance or examples on how a newcomer to Golang and Terraform can create a production grade provider. The topics I wanted more detail on were the Terraform schema, provider helpers, acceptance testing, provider package structure, development and documentation best practices etc.
There is a great section on extending Terraform that covers many of the intricacies of developing a provider and the scaffolding that Hashicorp provide to make it easier. As a newcomer I found it nearly incomprehensible the first time I looked at it. It assumes a good knowledge of Go and leaves many aspects of the Terraform helper libraries to the developer to work out for themselves. It is not an easy place to start.
This blog series seeks to provide an easier introduction to developing a robust and mature provider that can be released publicly with confidence. It comes out of researching answers and problem solving as I developed an extension to the IBM Cloud Terraform provider to support the IBM Cloud Internet Services offering (Cloudflare). I hope it will help those that want to understand how to go further in developing and releasing your own Terraform providers.
Basic provider examples are two a penny
I found many articles on how to create a provider. These were valuable to get started, to understand the structure of provider plugins and to score an early win by creating your own provider in Golang. In an hour or two you can have something working. There are many articles, but I found these helpful:
- Julien Fabre’s Writing a Terraform provider
- Peter Souter’s Writing and playing with custom Terraform Providers
- Write you own Terraform provider by Carlos Leon
The learning curve with Go and the Terraform provider framework is steep. Using the examples as a starting point, I created an MVP standalone provider that implemented Create, Read and Delete functions within a week. This basic provider can be found in my Github repo. In hindsight I now estimate this was only 20% of the overall effort in releasing the public version. Everyone will go through this stage.
Releasing a public provider
It is one thing to create a standalone provider for personal use, but what if you want to contribute to an existing provider? Or a provider that will be publicly released on GitHub and used in a myriad of uncharted ways. Testing to ensure that the provider works as expected throughout the Terraform resource lifecycle is a significant effort. Creating a robust and mature provider, along with the associated documentation and testing I found took at least 4 times longer than developing the initial MVP.
The terraform.io website includes a section on extending Terraform. Here you can find a detailed overview of the provider helpers, functions and test framework. This is where the hard work then starts of how you apply it to your development efforts.
Future topics
As I have learnt from others, I felt it was only appropriate to share my experience of provider development. Some of the topics I will explore in this series of blogs are:
- Terraform provider structure. A look at the key components and folders that make up the source of a provider on Github.
- Terraform Schema and the cloud resource’s API JSON data structures. Including how flatteners and expanders are used to convert between data formats.
- Client library development. Vendoring and Behaviour Driven Development (BDD) of the client API library in Go.
- Testing through the Terraform resource lifecycle. How to construct tests that exercise the full lifecycle.
I hope this exploration will help those that want to understand how a basic MVP provider can be matured into a public release with a lot less effort than my experience.
