DEW #2: Unbundling Data Science Workflows with Metaflow and AWS Step Functions

Ananth Packkildurai
Data Engineering Weekly
1 min readSep 9, 2020

Netflix open sourced metaflow.org December 2019. Metaflow follows a layered architecture approach to run the data workload, a contrasting approach from a tightly coupled airflow’s scheduler architecture. In this post, Netflix explains how the scheduler layer integrated with the AWS step functions.

--

--

Ananth Packkildurai
Data Engineering Weekly

Data Engineer. I write data engineering weekly; the weekly newsletter focused on data engineering. Subscribe at www.dataengineeringweekly.com.