The End of ETL As We Know It

If you’re as sick of this three-letter phrase as I am, you’ll be happy to know there is another way.

Paul Singman
Jan 20 · 4 min read
Image for post
Image for post

Take a Look Around You…

If you work in data in 2021, the acronym ETL is everywhere.

Replacing ETL with Intentional Data Transfer

The path forward is with ITD or Intentional Transfer of Data. You see, the need for ETL arises because no one builds their user database or CMS with downstream analytics in mind.

Image for post
Image for post
Example IDT architecture on AWS with a real-time Lambda consumer + durable storage to S3 | Image by author

1. IDT Forces Upfront Agreement on a Data Model Contract

How many times has one team changed a database table's schema, only to later learn the change broke a downstream analytics report? Any analytics veteran will tell you it’s a data tale as old as time!

{
"event_name": "transaction",
"user_id": 12345,
"event_action": "purchase",
"action_object": "gift_card",
"event_timestamp: "2021-01-02T03:04:05+01:00",
...
}

2. IDT Removes Data Processing Latencies

Most frequently, ETL jobs are run once-per-day overnight. But I’ve also worked on projects where they’ve run incrementally every 5 minutes. It all depends on the requirements of the data consumers.

Taking The First Steps

Moving from ETL to IDT isn’t a transformation that will happen for all your datasets overnight. Such an all-encompassing change would be overwhelming.

Whispering Data

Softly sharing the best kept analytics and productivity secrets

Sign up for Whispering Data

By Whispering Data

Whispering Data is a publication for all the data & productivity secrets you wish you knew years ago! Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Paul Singman

Written by

ML Engineering Lead at Equinox. Whisperer of data and productivity wisdom. Standing on the shoulders of giants.

Whispering Data

Whispering Data is a Medium publication for all the data & productivity secrets you wish you knew years ago!

Paul Singman

Written by

ML Engineering Lead at Equinox. Whisperer of data and productivity wisdom. Standing on the shoulders of giants.

Whispering Data

Whispering Data is a Medium publication for all the data & productivity secrets you wish you knew years ago!

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface.

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox.

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic.

Get the Medium app