Data Engineering Digest #10 (March 2020)

Maycon Viana Bordin

Published in

data.plumbers

11 min readApr 10, 2020

Data Engineering Role

Doing Data in a “normal” company

A personal and honest review

towardsdatascience.com

Courses & Training

Top 6 Courses to learn Big Data and Hadoop in 2020

Hello guys, If you are learning Hadoop or Big Data and are looking for some awesome courses, then you have come to the…

medium.com

Prep for Databricks Exam 3a : DataFrameReader

As I walk through the Databricks exam prep for Apache Spark 2.4 with Python 3, I’m collating notes based on the…

medium.com

Podcasts & Presentations

The Life Of A Non-Profit Data Professional

An interview with Tyler Colby about his experiences working as a data professional in the non-profit sector and the…

www.dataengineeringpodcast.com

Behind The Scenes Of The Linode Object Storage Service

An interview with the project lead for Linode's recently released object storage service about the challenges involved…

www.dataengineeringpodcast.com

Building A New Foundation For CouchDB

An interview about the CouchDB document database and the work being done to rearchitect it to run on top of…

www.dataengineeringpodcast.com

Scaling Data Governance For Global Businesses With A Data Hub Architecture

An interview about how a data hub architecture can reduce the overhead of managing data governance and compliance…

www.dataengineeringpodcast.com

Easier Stream Processing On Kafka With ksqlDB

An interview about the ksqlDB platform and the unified experience that it provides for building stream processing…

www.dataengineeringpodcast.com

Monitoring and Tracing @Netflix Streaming Data Infrastructure

Allen Wang talks about the design and implementation details of the dev/ops tools used by Netflix and highlights the…

www.infoq.com

Kafka Needs No Keeper

Colin McCabe talks about the ongoing effort to replace the use of Zookeeper in Kafka: why they want to do it and how it…

www.infoq.com

Real Data Architectures

Architecture for High-Throughput Low-Latency Big Data Pipeline on Cloud

Scalable and efficient data pipelines are as important for the success of analytics and ML as reliable supply lines are…

towardsdatascience.com

Drop’s Data Lake

Drop Engineering’s journey to building a scalable, and server-less data lake.

drop.engineering

Loading and transforming data to Big Query at large scale

Data Lake Platform

medium.com

The Story of Building the Datalake at Varo

What is a Data Lake? What problem are we solving by building a Data Lake? Please note that throughout this blog, we…

medium.com

Building a marketing data lake using AWS: A Commandeer App Case Study — Part 1 of 11

By: Sergeant Serverless

medium.com

Real-time Stream Analytics and User Scoring Using Apache Druid, Flink & Cassandra at Deep.BI

At Deep.BI we track user habits, engagement, product & content performance — processing up to terabytes or billions of…

medium.com

Load Data From MySQL(Cloud SQL) To BigQuery Using Embulk (+ AirFlow Scheduling)

Welcome! 😄 I’m JooYoung Lim, Data Engineer in YPLabs Co.,

medium.com

Structuring a Robust Data Pipeline

We all know and love data. Data holds insight. Data helps you make decisions. But data is a dog (pun here)… things can…

medium.com

Data Culture

A Transformation Journey for Enabling the Data-driven Enterprise

Where does Data transformation sit in any context of Transformation today? Why Data gets left out during a Digital…

medium.com

Data Lake

A complete guide about how to break the data monolith.

How to deal with the new big goal in the software world.

medium.com

Architecting Modern Data Platforms

Welcome to my post on ‘Architecting Modern Data Platforms’, here I will be discussing how to design cutting edge data…

medium.com

Why and When to Avoid S3 as a Data Platform for Data Lakes

Data lakes are all the rage these days in large enterprises. A data lake is a single store for both raw copies of…

towardsdatascience.com

Data Lakes, Data Warehouses & Data Hubs

How to combine them ?

medium.com

Data partitioning: good practices in the design of Data Lakes.

Why Partitioning?

medium.com

The Future of Data Lakes

A fine line exists between pristine data lakes and data swamps. For many, lakes have become too complex. A new data…

medium.com

Data Governance

Data quality impact on the dataset

We look at the different aspects of data quality that impact on the dimensions of our dataset, and the importance of…

towardsdatascience.com

Starting A Data Quality Checklist

Go through this list before starting a project based on data.

medium.com

Data Formats

Are We Taking Only Half Of The Advantage Of Columnar File Format?

Sorting the records in columnar data format is a critical design considerations that many of us have not paid…

medium.com

Reshape Data Lake: Delta, Iceberg, Hudi, or Hive

The super success of Spark in the ETL area also showed that many paradigms in the traditional data warehouse are indeed…

medium.com

Compressing a year of Reddit with Apache Avro and Google Protobuf

Big data projects have always seemed to generate a lot of hype and excitement among developers. And for good reason…

medium.com

Delta Lake

DataBricks Delta Lake

Databricks Delta Tables — Upsert — Merge Error

medium.com

Databricks Delta — Partitioning best practice

Partitioning is good …and bad

medium.com

Data Pipelines

Don’t use SQL for your Big Data ELT pipelines

The time of coding data pipelines with high-level programming languages

medium.com

Data Engineering 101: Writing Your First Pipeline

In Airflow and Luigi

medium.com

Data Quality Tools

Data Sentinel: Automating data validation

Data's value is best realized when prepared and treated correctly. However, when you're working with data at an…

engineering.linkedin.com

Data Processing

Guide to Big Data Joins - Python, SQL, Pandas, Spark, Dask

To find the top average ratings for movies you need to join links to metadata to ratings: SELECT m.title, avg(r.rating)…

towardsdatascience.com

Apache Spark

On Spark, Hive, and Small Files: An In-Depth Look at Spark Partitioning Strategies

One of the most common ways to store results from a Spark job is by writing the results to a Hive table stored on HDFS…

medium.com

Spark Dynamic Partition Inserts — Part 1

By: Roi Teveth and Itai Yaffe

medium.com

How to build Spark from source and deploy it to a Kubernetes cluster in 60 minutes

Get on the hype train for Big Data in Kubernetes with this Spark tutorial

towardsdatascience.com

Spark Dynamic Partition Inserts and AWS S3 — Part 2

By: Roi Teveth and Itai Yaffe

medium.com

Chaining Multiple MapReduce Jobs with Hadoop/ Java

I learned about MapReduce briefly pretty much a year ago when my job required a bit of Hadoop. I then had not touched…

towardsdatascience.com

Advanced schema management for Spark applications at scale

With the scale of LinkedIn's data processing ecosystem, it is mandatory to provide a solution that imposes minimal…

engineering.linkedin.com

Apache Hive

Building a Data warehouse with Hive at Helpshift — Part 1

This will be a series of posts describing how we built Helpshift’s data warehouse and how we run it on production. It…

medium.com

Apache Hive and Apache Impala- What you should be knowing?

When we want to perform more data-intensive tasks, we leverage Hive. For tasks related to querying, processing…

medium.com

Presto

Presto with Kubernetes and S3 — Deployment

I have been working in the big data arena for more than ten years. If you ask me what is the most popular use case in…

medium.com

A Presto Data Pipeline with S3

This post presents a modern data warehouse implemented with Presto and FlashBlade S3; using Presto to ingest data and…

medium.com

Stream Processing

Stream Processing Fundamentals

Introduction to Stream Processing

medium.com

Data Engineering 101: From Batch Processing to Streaming

How does data get to the data scientists and machine learning engineers anyway?

medium.com

Apache Beam

Create your first ETL Pipeline in Apache Beam

Learn how to use Apache Beam to create efficient Pipelines for your applications.

towardsdatascience.com

Apache Flink

Flink on Zeppelin — (Part 1): Get Started

Introduction

medium.com

Deploy Flink Jobs on Kubernetes

Learn how to build a Flink Cluster using Kubernetes and deploy a custom job

medium.com

How to rule the chaos with Apache Flink?

Real-time anomaly detection for microservice monitoring with Apache Flink

medium.com

Apache Flink Specifying Keys

KeyBy is one of the mostly used transformation operator for data streams. It is used to partition the data stream based…

medium.com

Windowing in Flink

Windowing is a crucial concept in stream processing frameworks or when we are dealing with an infinite amount of data…

medium.com

Application Log Intelligence & Performance Insights at Salesforce using Flink

How Salesforce uses Apache Flink for application log intelligence, real-time anomaly detection and performance…

engineering.salesforce.com

Apache Flink State Types

Apache Flink is 4th generation open source data processing framework. Flink does support stateful and stateless…

medium.com

Apache Spark Streaming

How to Monitor Data Stream Quality Using Spark Streaming and Delta Lake

Try this notebook to reproduce the steps outlined below In the era of accelerating everything, streaming data is no…

databricks.com

Knowing PySpark and Kafka: A 100 Million Events Use-Case

A typical day in the eCommerce market of streaming 10,000 events per minute and choosing the right tools to handle it

towardsdatascience.com

Siddhi

Siddhi MongoDB connector optimized to increase performance upto 4 times.

Siddhi is a cloud native Streaming and Complex Event Processing engine which can understand streaming SQL queries…

medium.com

Apache Kafka Streams

Learn Stream Processing With Kafka Streams

Stateless operations

medium.com

Getting started with Kafka Streams

With growing volumes of data (statistics says data gets doubled every 2–3 years),its acquisition and its subsequent…

medium.com

How to Test Kafka Streams Applications

Validating topologies through testing

medium.com

How to Use Stateful Operations in Kafka Streams

Explore stateful operations in the Kafka Streams DSL API

medium.com

Change Data Capture

How to Build your First Real-Time Streaming(CDC) system(Introduction-Part 1)

Introduction

-Part 1) Introductionmedium.com

Debezium

Transaction Log Tailing With Debezium - Part 2:

In the previous post of this post series, I explained what Transactional Log Tailing and Debezium are and why we need…

medium.com

PostgreSql Db Change Data Capture (CDC) Using Debezium

It’s clear that all kind of data is very important for today’s software environment. Especially end-user oriented data…

medium.com

Log based CDC between DB2 & MySQL using Debezium

A quick pipeline setup using Debezium to setup a whole CDC pipeline from DB2 to MySQL.

medium.com

Messaging

Apache Kafka

Apache Kafka — An Introduction

Before we begin, the picture below shows the list of companies that are currently using Apache Kafka. As you can see…

medium.com

Facts about Kafka every business should know

A short guide on Apache Kafka for every team that’s considering this stream processing platform.

blog.softwaremill.com

Protobuf as an encoding format for Apache Kafka

When talking about schema in the context of Apache Kafka, Apache Avro is the de facto standard. But what about Google…

medium.com

Kafka Acks Explained

Visualizing Kafka’s most misunderstood configuration setting

medium.com

Help, Kafka ate my data!

How to prevent data loss in Kafka - the most important configuration explained.

blog.softwaremill.com

Ingest, Store and Search JSON data with Apache Kafka and Apache Geode

We’ll be ingesting JSON data from a rest endpoint into a Kafka topic and sink the data into an Apache Geode region

medium.com

Apache Pulsar

Integration testing Apache Pulsar clients with Docker Compose and TestContainers

How to enable connectivity between multiple Docker Compose environments when using the TestContainers library with…

medium.com

Workflow Management

Apache Airflow

Improving performance of Apache Airflow scheduler

Apache Airflow is an open-source tool to create and manage complex workflows. More recently, Airflow has been gaining a…

medium.com

You may not need Airflow…. yet

Airflow is robust and flexible, but complicated. You may want to try more tailored solutions such as Stitch, DBT…

towardsdatascience.com

A gentle introduction to Data Workflows with Apache Airflow and Apache Spark

Imagine you’d developed a transformation process in a local Spark and you want to schedule it so a simple Cron Job…

medium.com

Deploying Apache Airflow to Google Kubernetes Engine

The Easy Way using Bitnami Helm Charts

medium.com

Scale Your Data Pipelines with Airflow and Kubernetes

The Perfect Airflow Setup

towardsdatascience.com

End to End tests using Apache Airflow

Apache Airflow is a platform created by community to programmatically author, schedule and monitor workflows. You…

medium.com

Apache Airflow: Manage thousands of DAGs

Controlling and managing your DAGs are some of the most important things in a scheduler platform. Which best solutions…

medium.com

Apache Airflow — Crash overview

Data pipelines are data processing functions or different tasks put together in a pipelined fashion. For example, you…

medium.com

Building a resilient Anomaly Detection pipeline with Apache Airflow

Airflow is known for its powerful scheduling and orchestration capabilities .This article assumes that you already have…

medium.com

Prefect

Open-Sourcing the Prefect Platform

Today, we’re taking a huge step forward and open-sourcing an orchestration layer for Prefect — including the Prefect…

medium.com

Cloud Providers

AWS

AWS Glue and You

In this post, I will walk through the architecture and implementation involved in a cloud ETL (extract, transform…

towardsdatascience.com

Creating Stream Processors with AWS Lambda Functions

There Can Be More Than One

medium.com

Using AWS DMS to tail MongoDB

AWS DMS as a streaming data ingestion tool rather than just a one time migration tool.

towardsdatascience.com

Glue Crawler optimization/alternative | Athena use case

The aim of this blog is to see different alternatives to Glue Crawler in order to query data from s3 using AWS Athena…

medium.com

How to use Glue to load data from S3 into an AWS Elasticsearch cluster that’s in your VPC

I’m writing this because when I had this problem, there didn’t seem to be any resources online detailing how to solve…

levelup.gitconnected.com

Pivot table in AWS Athena (or Presto)

Background

medium.com

DynamoDB on Production: A compilation of lessons learned in a year

This article talks about the do’s, don’ts, and best practices for using DynamoDB for large scale production workloads…

medium.com

Google Cloud

Model deployment with Apache Beam and Dataflow

An option for quick and simple operating your analytical models with Apache Beam / Dataflow, BigQuery, and Scikit-learn

towardsdatascience.com

Data ingestion Google Big Query without the headaches

Schema conversions on the fly without the headaches with Dataflow and BigQuery.

blog.doit-intl.com

Azure

Process Azure Analysis Services Models with Azure Data Factory v2

Azure Data Factory (ADFv2) is Microsoft’s cloud-based data movement, integration and transformation service. It offers…

medium.com

Manage CI/CD pipelines using Azure DevOps, Azure Data Factory & Azure Databricks

In Part 1 of this post, I will show you the process of automating a data factory pipeline using Azure DevOps.

medium.com

Replicating changes from SQL Managed Instance to the Data Lake using Change Data Capture and Azure…

Introduction

medium.com

Azure Data Factory — Retrieve Key Vault secrets in your pipelines at runtime

Azure Data Factory and Azure Key Vault: better together

medium.com

Power BI Connects to Azure Databricks

Deriving the Spark URL and creating user token that is used by PowerBI to authenticate into Azure Databricks.

towardsdatascience.com

How to Increase Azure Databricks Cluster vCPU Cores Limits

The solution of the warning: “This account may not have enough CPU cores to satisfy this request” for Azure Databricks

towardsdatascience.com

The Secret behind Databricks and Azure Data Lake — I

Azure Databricks, Azure KeyVault, Azure DataLake Storage

medium.com

Databases

NoSQL

Analytics for billions of records — Part 1

At Moka, we believe that data is one of the important resources for empowering businesses to grow. Thousands of our…

medium.com

How Netflix uses Druid for Real-time Insights to Ensure a High-Quality Experience

By Ben Sykes

netflixtechblog.com

It’s Time to Familiarize Yourself With NoSQL Databases More Than Ever

SQL vs NoSQL: the difficult choice of the right database (and how to do it right)

towardsdatascience.com

KeyDB as a [possible] replacement for Redis

Our experience with a multi-threaded Redis alternative

medium.com

Migrating from Cassandra to ScyllaDB

What is Scylla?

medium.com

MongoDB: Maintain Data Integrity with Atomic Operations

A closer look at MongoDB’s $set, $setOnInsert, $push, $pull and $inc update operators

levelup.gitconnected.com

Bucketisation: Using cassandra for time series data scans.

Coming from RDBMS world it takes some getting used to before you can efficiently start modelling entities in Cassandra…

medium.com

Apache Druid — The sine qua non of contemporary Big-Data analytics

The term “big data” refers to the digital stores of data that have a high volume, velocity and variety. Big data…

medium.com

All you need to know about Apache Kudu

In this article I will describe and simplify everything you should know about Apache Kudu since you already know the…

medium.com

How We Pushed a Million Keys to Redis in Seconds

Dealing with a lot of keys? Redis’ Pipe Mode is your friend.

blog.gojekengineering.com

In-Memory & Data Grid

Apache Ignite — More Than Just a Data Grid

medium.com

Apache Ignite on Kubernetes: Things to Know About

Kubernetes is becoming more and more popular these days. No wonder! It’s an awesome technology that helps a lot in…

medium.com