Data Engineering Digest #9 (February 2020)

Photo by Pixabay from Pexels

New Tools

Five Interesting Data Engineering Projects

Python ETL Tools: Best 8 Options

Data Engineering Role

When Data Science Meets Data Engineering

How to Become a Data Engineer | Springboard India Blog

Events

Official Spark + AI Summit 2020 Keynote Speaker List

Podcasts & Presentations

Data Infrastructure Automation For Private SaaS At Snowplow

Shining A Light on Shadow IT In Data And Analytics

The Benefits And Challenges Of Building A Data Trust

Data Modeling That Evolves With Your Business Using Data Vault

Data Mesh Paradigm Shift in Data Platform Architecture

ZooKeeper Meetup@Facebook: Advancing the state of distributed coordination

Publications

Experiences with Managing Data Ingestion into a Corporate Datalake

We explain our experiences in designing, building and running a large corporate Datalake. Our platform has been running for over two years and makes a wide variety of corporate data assets, such as sales, marketing, customer information, as well as data from less conventional sources such as weather, news and social media available for analytics purposes to many teams across the company. We focus on describing the management of data and in particular how it is transferred and ingested into the platform.

Real Data Architectures

Building an Adaptive, Multi-Tenant Stream Bus with Kafka and Golang

How AppsFlyer uses Apache Airflow to run over 3.5k daily jobs and more

Data Culture

Speed up your Data Transformation with Data Catalysts

Building a Data-Driven Culture from the Ground Up

Generalists vs. Specialists in Data Science and Analytics

Data Lake

What is the Data Architecture we Need?

Data Governance

Data Access Governance for JupyterHub on AWS with Privacera

Towards a Data Quality Score in open data (part 2)

How We Improved Data Discovery for Data Scientists at Spotify

Data Catalogue — Knowing your data

Event Sourcing

Journey to Event Driven — Part 1: Why Event-First Thinking Changes Everything

Data Formats

Modern Data Lakes Overview

Apache HUDI vs Delta Lake

RION - A Fast, Compact, Versatile Data Format - DZone Big Data

Delta Lake

Simplifying Data Ingestion with Auto Loader for Delta Lake

Data Pipelines

A Simple Data Lake ETL Pipeline

Data Processing

Apache Spark

Cache Patterns with Apache Spark

Building a Scalable Spark cluster with Docker Containers

Ignite the Spark!

Data Sharing between multiple Spark Jobs in Databricks

Enhancing Optimized PySpark Queries

Boosting Apache Spark with GPUs and the RAPIDS Library

Using Spark Listeners - DZone Big Data

Apache Hadoop

Hadoop: Decade Two, Day Zero*

Hadoop & The Raspberry Pi: My Final Year Project

MR3

Announce: MR3 1.0 released

Stream Processing

Apache Flink

Application Log Intelligence & Performance Insights at Salesforce using Flink

Apache Flink Architecture Overview

Scaling Page Scraping with Apache Flink

Deep Dive Into Apache Flink's TumblingWindow - Part 1 - DZone Big Data

Apache Spark

Building a real-time prediction pipeline using Spark Structured Streaming and Microservices

Change Data Capture

Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline

Grafana Dashboard For Monitoring Debezium MySQL Connector

Ensuring data consistency across services with the Transactional Outbox pattern

Streaming SQL Server CDC with Apache Kafka using Debezium.

Moving from Hibernate Envers to Debezium

Distributed Data for Microservices — Event Sourcing vs. Change Data Capture · Debezium

Messaging

Why Apache Pulsar — A Gentle Comparison with Kafka

Apache Kafka

Why Kafka Is so Fast

Apache Kafka with Kubernetes — Provision and Performance

Processing Large Messages with Kafka Streams

SIEM with Osquery Event Log Aggregation and Confluent Platform

Workflow Management

Comparison of pipeline/workflow packages: Airflow, Luigi, Gokart, Metaflow, Kedro, PipelineX

Apache Airflow

How to deploy Apache Airflow with Celery on AWS

Data Pipeline using Apache Airflow

Data Quality Libraries

Usage of Deequ Suite by Amazon for DQC(Data Quality Checks) for your data

Data Profiling using Spark & Deequ

Cloud Providers

Google Cloud Advantages over AWS

AWS

Overview of Data Analytics in AWS — Glue, Athena and DataLake

An Upgrade: Part 3 — Querying a Data Lake in AWS with Amazon Athena

Amazon Redshift CI/CD - How we did it and why you should do it too

Tutorial : Building a S3-Parquet DataLake without a single line of Code !

Data Engineering: Data transformation using AWS cloud formation, Apache Airflow, Apache Livy and…

Streaming ETL with Apache Flink and Amazon Kinesis Data Analytics | Amazon Web Services

How Siemens built a fully managed scheduling mechanism for updates on Amazon S3 data lakes | Amazon…

Load data incrementally and optimized Parquet writer with AWS Glue | Amazon Web Services

Azure

Working with Azure Cosmos DB in your Azure Functions

Databases

NoSQL

Migrating to CockroachDB

Kudu PART 1

Apache Kudu series: 2. Troubleshooting for TABLET_DATA_TOMBSTONED

Apache Kudu series: 3. Troubleshooting:- Add or remove data directories.

Apache Druid Production Setup in Google Cloud Platform with Dataproc cluster — Part 1

Experiences with Tombstones in Apache Cassandra

Storage

Benchmarking Ozone: Cloudera’s next generation Storage for CDP — Cloudera Blog

Modern Data Warehouses

Building a Data Warehouse: Basic Architectural principles

Introducing Materialize: the Streaming Data Warehouse