Uncanny Recursions
Published in

Uncanny Recursions

Building an end-to-end data pipeline from scratch

This series is meant to cover a broad range of topics that involve setting up a production grade ETL pipeline. I have broken it down into chapters and more of them will be added as we move along this series.

I am a Mac user so everything that is written is in context of software and tools installed on OS X. You should be able to find and install equivalent versions of those for your own setup.

Feel free to leave any feedback on my Twitter handle or through my website.

Chapter 1 - Orchestration basics: Setting up Airflow in a local Kubernetes cluster using helm

Chapter 2 - Introduction to KubernetesExecutor and KubernetesPodOperator

--

--

--

A software engineer’s journey around tech and product

Recommended from Medium

Russian Peasant Multiplication — with Elixir — Part 2

Russian Peasant Multiplication — with Elixir Part 2

A practical example of using yield in C# in 2022

Type-Safe & Composable GraphQL in Elm

Why Cloud Cost Optimization Shouldn’t be Finance’s Responsibility

How to create a dynamic development environment using Kubernetes namespaces and ArgoCD

Apache ShardingSphere Enterprise Applications

Evolution of Windows OS

How to have different directories for .git and working-tree?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Rafay Aleem

Rafay Aleem

Data Engineer at Faire. Based in Toronto. Music aficionado who likes playing guitar and is an Eric Clapton fan.

More from Medium

Analytics and Datawarehouse Implementation of Global Terrorism Database

DevOps in Data: The Volcanic Waltz

A volcano erupting

FluentD webhook with TLS Mutual Authentication

Big data and the Cloud: A Blog about the Implementation of Big Data in the Cloud