Top stories published by Data Engineering Weekly in 2020

Homepage

Open in app

Top Stories published by Data Engineering Weekly in 2020

All

2020

January

February

March

April

May

June

July

August

September

October

November

December

Ananth Packkildurai in Data Engineering Weekly

Dec 27, 2020

Back To The Future: Data Engineering Trends 2020 & Beyond

This article was originally published on dataengineeringweekly.com. Subscribe to the data engineering weekly newsletter to know the latest development in data engineering. Subscribe to Data Engineering Weekly here.

1 response

Ananth Packkildurai in Data Engineering Weekly

Dec 20, 2020

Data Engineering Weekly #22

This is the cross-post of Data Engineering Weekly.

Subscribe https://www.dataengineeringweekly.com/ for the latest update.

Welcome to the 22nd edition of the data engineering newsletter. This week’s release is a new set of…

Ananth Packkildurai in Data Engineering Weekly

Dec 13, 2020

Data Engineering Weekly #21: Metadata Edition

This is the cross-post of Data Engineering Weekly. Subscribe https://www.dataengineeringweekly.com/ for the latest update.

Welcome to the 21st edition of the data engineering newsletter. The 21st edition of the newsletter…

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #1: Sharing is caring — Catalogs in Flink SQL

Catalog services are an essential metadata engine for data discovery and schema management. Hive meta store, AWS Glue data catalog are some of the catalog services used in data infrastructure. Apache Flink 1.9 added catalog integration and this…

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #2: Byte Down: Making Netflix’s Data Infrastructure Cost-Effective

Cost optimization is an essential engineering in cloud computing. Netflix writes about its cost optimization platform in this blog post. The automated TTL recommendations only for tables with material…

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #1: Streaming with Spring Cloud

Walmart wrote about stream processing with Spring Cloud. Spring Cloud provides stream processing on top of the familiar spring framework. The post gives an introduction to Spring Cloud, a sample application, and how to unit test.

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #2: Airflow Functional DAG Definition

The Airflow operator represents a single idempotent task. Operators determine what executes when your DAG runs. One of the drawbacks of the operator is that no Airflow does not have explicit inter-operator communication, aka no easy way to pass messages…

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #2: Empowering Pinterest data scientists and machine learning engineers with PySpark

Python becomes the de facto language for data science workload. Apache Spark community continually improves the performance of PySpark. Pinterest writes about its data…

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #2: Unbundling Data Science Workflows with Metaflow and AWS Step Functions

Netflix open sourced metaflow.org December 2019. Metaflow follows a layered architecture approach to run the data workload, a contrasting approach from a tightly coupled airflow’s scheduler…

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #1: How We’re Solving Data Discovery Challenges at Shopify

Data discoverability is an essential aspect of the data infrastructure. The value proportion of a data warehouse system exponentially decreases with a weak data discovery system. The Shopify data team writes about their…

2021

These were the top 10 stories published by Data Engineering Weekly in 2020. You can also dive into monthly archives for 2020 by using the calendar at the top of this page.

About

Data Engineering Weekly

Weekly data engineering newsletter. Subscribe to https://www.dataengineeringweekly.com. Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent any of my employer’s opinion.

More information