Programmatically Managed Airbyte using Terraform

Photo by Qijin Xu on Unsplash

Airbyte is an open-source data pipeline platform that serves as an alternative to Stitch Data and Fivetran. Though existing data pipeline platforms offer a significant number of integrations with well-regarded sources like Stripe and Salesforce, there is a gap in the current model that leaves out small service integrations.

Terraform is a tool for provisioning infrastructure (or managing Infrastructure as Code). It supports almost all the cloud providers like AWS, Google Cloud, Azure, etc. Seamless integration of Terraform cloud with Terraform allows you to use Terraform to integrate with various CI-CD tools.

Well, what if we combine Airbyte and Terraform to achieve the programmatically managed Airbyte using Terraform so we can unlock some of these benefits:

• Save time managing ELT & accelerate your testing cycle
• Version-control & automate your ELT infrastructure
• Migrate your Airbyte resources with one line of code

Goals

The best way to learn something new is start it with the simplest one.

I will not overcomplicate everything, so the goal of this article is to create Sources, Destinations, and Connections in Airbyte using Terraform. We will use Faker data as the source from the…