Top stories published by Data Engineering Weekly in September of 2020

Homepage

Open in app

Top Stories published by Data Engineering Weekly in September of 2020

All

2020

September

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #1: Sharing is caring — Catalogs in Flink SQL

Catalog services are an essential metadata engine for data discovery and schema management. Hive meta store, AWS Glue data catalog are some of the catalog services used in data infrastructure. Apache Flink 1.9 added catalog integration and this…

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #2: Byte Down: Making Netflix’s Data Infrastructure Cost-Effective

Cost optimization is an essential engineering in cloud computing. Netflix writes about its cost optimization platform in this blog post. The automated TTL recommendations only for tables with material…

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #1: Streaming with Spring Cloud

Walmart wrote about stream processing with Spring Cloud. Spring Cloud provides stream processing on top of the familiar spring framework. The post gives an introduction to Spring Cloud, a sample application, and how to unit test.

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #2: Airflow Functional DAG Definition

The Airflow operator represents a single idempotent task. Operators determine what executes when your DAG runs. One of the drawbacks of the operator is that no Airflow does not have explicit inter-operator communication, aka no easy way to pass messages…

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #2: Empowering Pinterest data scientists and machine learning engineers with PySpark

Python becomes the de facto language for data science workload. Apache Spark community continually improves the performance of PySpark. Pinterest writes about its data…

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #2: Unbundling Data Science Workflows with Metaflow and AWS Step Functions

Netflix open sourced metaflow.org December 2019. Metaflow follows a layered architecture approach to run the data workload, a contrasting approach from a tightly coupled airflow’s scheduler…

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #1: How We’re Solving Data Discovery Challenges at Shopify

Data discoverability is an essential aspect of the data infrastructure. The value proportion of a data warehouse system exponentially decreases with a weak data discovery system. The Shopify data team writes about their…

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #1: A Comprehensive Look at Dates and Timestamps in Apache Spark™ 3.0

TimeZone is a complicated yet crucial part of data infrastructure. Databricks writes an excellent overview of TimeZone, Dates, and Timestamp with Spark 3.0

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #2: Tasted Apache Pinot and we Loved it!

Apache Pinot is gaining momentum as a realtime OLAP system for data engineering needs. In this blog post, Sapient narrates its experience benchmarking Apache Pinot. The ingestion rate cross 120k entries/second on one node is impressive.

Ananth Packkildurai in Data Engineering Weekly

Sep 8, 2020

DEW #1: Driving Shopping Upsells from Pinterest Search

Pinterest writes shopping intent ML model to drive the shopping upsells Pinterest search. The evolution of the model from the upsell click rate model to the “long click” model is an exciting read.

December 2020

These were the top 10 stories published by Data Engineering Weekly in September of 2020. You can also dive into daily archives for September of 2020 by using the calendar at the top of this page.

About

Data Engineering Weekly

Weekly data engineering newsletter. Subscribe to https://www.dataengineeringweekly.com. Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent any of my employer’s opinion.

More information