Overview Over the past two years, the Devoted Health Data Engineering team has evolved its data architecture to support the needs of a rapidly growing business with increasing data volumes. …

What is Snowflake? Snowflake is a managed cloud data warehouse solution. It is similar to BigQuery or Redshift, but has some unique features like separation of compute and storage and strong support for semi-structured data (JSON, Parquet, Avro) that differentiate it. What is Terraform? Terraform is a tool from HashiCorp for managing infrastructure via code. You…

Apache Airflow is an open-source workflow orchestration tool. There are many posts available that explain the core concepts of Airflow (I recommend this one). This post assumes you have some familiarity with these concepts and focuses on how we develop, test, and deploy Airflow and Airflow DAGs at Devoted Health…

Apache Airflow is “a platform to programmatically author, schedule, and monitor workflows.” And it is currently having its moment. At DataEngConf NYC 2018, it seemed like every other talk was either about or mentioned Airflow. …