What is Snowflake?
Snowflake is a managed cloud data warehouse solution. It is similar to BigQuery or Redshift, but has some unique features like separation of compute and storage and strong support for semi-structured data (JSON, Parquet, Avro) that differentiate it.
What is Terraform?
Terraform is a tool from HashiCorp for managing infrastructure via code. You can use it to provision, update, or delete a wide range of resources like EC2 instances, Datadog monitors, OpsGenie schedules and more. This allows you to know the current state of your infrastructure at any given time and to control how it is updated. Terraform ships with many providers. Unfortunately…
Apache Airflow is an open-source workflow orchestration tool. There are many posts available that explain the core concepts of Airflow (I recommend this one). This post assumes you have some familiarity with these concepts and focuses on how we develop, test, and deploy Airflow and Airflow DAGs at Devoted Health. Devoted is a Medicare Advantage startup aimed at making healthcare easier, more affordable, and believes every member should be treated like we would treat a member of our own family.
Airflow Deployment
This part of the post discusses Kubernetes, Helm, Terraform, and Docker, but since they are all their own complicated things…
Apache Airflow is “a platform to programmatically author, schedule, and monitor workflows.” And it is currently having its moment. At DataEngConf NYC 2018, it seemed like every other talk was either about or mentioned Airflow. There have also been countless blog posts about how different companies are using the tool and it even has a podcast!
A major use case for Airflow seems to be ETL or ELT or ETTL or whatever acronym we are using today for moving data in batches from production systems to data warehouses. This is a pattern that will typically be repeated in multiple pipelines…
Adam
Data Engineer at Devoted Health
