Data Engineering Digest #7 (December 2019)Maycon Viana Bordin·FollowPublished indata.plumbers·8 min read·Jan 4, 2020--ListenSharePhoto by Peter Unden from PexelsAWS re:Invent 20192019 re:Invent PageAWS re:Invent 2019 — Keynote with Andy Jassy AWS re:Invent 2019 — Keynote with Dr. Werner Vogels Monday Night Live with…aws.amazon.comNew ToolsToolbox for Spark, launching soonGigahex, the free toolbox for Spark applications will be available for developer’s preview in Q1 2020. The access will…medium.comIntroducing JindoFS: A High-performance Data Lake Storage SolutionJindoFS is a cloud-native file system that integrates the advantages of local disks and the ultra-large capacity of…medium.comCourses & TrainingHow to pass the Google Cloud Professional Data Engineer examWithout the recommended 3-year industry experiencetowardsdatascience.comPublicationsData Lakes: Trends and PerspectivesAs a relatively new concept, data lake has neither a standard definition nor an acknowledged architecture. Thus, we…hal.archives-ouvertes.frData Lake Architecture for a Banking Data Model5 Pages Posted: 11 Dec 2019 Date Written: September 12, 2019 Industry models provide an excellent opportunity to…papers.ssrn.comA framework for performance-oriented real-time stream processing over the cloudAbstract Stream processing is an in-memory computing paradigm that supports querying over continuous data streams with…minerva-access.unimelb.edu.auMerge, Split, and Cluster: Dynamic Deployment of Stream Processing ApplicationsStream Processing has become a major programming model to timely handle large volumes of data generated at the edge of…hal.inria.frData LakeData Mesh AppliedSubtitle: Moving step-by-step from mono data lake to decentralized 21st-century data mesh.towardsdatascience.comThree Reasons to Build a Security Data LakeSeveral years after the initial hype fizzled out, security data lakes are making a big (data) comeback. Here are three…medium.comData Lakes, and SQL???Not a paradox. SQL is being used for analysis and transformation of large volumes of data in data lakes.towardsdatascience.comData GovernanceMaster Data Management, how to match and merge records to unify your dataAt the heart of most data governance projects, lies the concept of MDM — Master Data Management — which aims at…medium.comDataOps, a growing trend in data engineeringFrom the need to deliver quality data quickly and continuously, emerges DataOps, an approach that promises agile data…medium.comA HandBook Dictionary on DataOps and Its ImportanceBig giants like Google and Amazon, release software quite often in a day! Reason? They started to implement DevOps…medium.comDeveloping with Data in KubernetesThe Titan Project helps developers manage their data like code. Run, version, and share data in Kubernetes with an easy…medium.comData QualityHow to do data quality with DataOpsLeveraging data tests and safe environments makes data quality an everyday activity for everyone who touches your datatowardsdatascience.comMaster Data Management, how to match and merge records to unify your dataAt the heart of most data governance projects, lies the concept of MDM — Master Data Management — which aims at…medium.comStrategies to effectively propagate Master Data MergesUnifying records into Master records is hard, but propagating their prove even harder.medium.comData FormatsGPU Datascience: Converting CSV to Parquet / ORC faster & Cheaper than using a cluster!At Locally something has rung true for a long time: “We deal with Enterprise levels of data on Start-up money” — we…towardsdatascience.comBig Data File Formats Explained Using Spark Part 1Understand How Avro, Parquet & ORC Workmedium.comCDCDBLog: A Generic Change-Data-Capture FrameworkAndreas Andreakis, Ioannis Papapanagiotoumedium.comRealTime CDC From MySQL Using AWS MSK With DebeziumCDC is becoming more popular nowadays. We are going to realtime CDC from MySQL to S3 using AWS MSK with Debezium…medium.comStream ProcessingStream Processing with SiddhiToday, data is produced in large scale. In fact, Quintilian bytes of data is produced in a day and it is ever growing…medium.comEvents and Data Processing : 10 Must Learn TopicsThere has been tremendous progress in event and large scale data processing (batch or stream) in the last two decades…medium.comHow to secure your Azkarra Streams application for productionAzkarra is a recent open-source framework which provides the necessary functionalities to secure your Kafka Streams…medium.comApache SparkThe Complete Apache Spark Collection [Tutorials and Articles] - DZone Big DataIn this edition of "Best of DZone," we've compiled our best tutorials and articles on one of the most popular analytics…dzone.comWrite Clean and SOLID Scala Spark Jobsnowadays extensive pipelines are written as simple SQL queries, neglecting important development concepts as writing…towardsdatascience.comWhy Apache Spark Is Fast and How to Make It Run FasterA deep dive into Apache Spark and how it functionstowardsdatascience.comThe art of joining in SparkPractical tips to speedup joins in Sparktowardsdatascience.comPySpark for Data Science WorkflowsChapter 6 of Data Science in Productiontowardsdatascience.comGetting Started with Data Analytics using Jupyter Notebooks, PySpark, and DockerDemonstrate the creation of a containerized data analytics environment using Jupyter Docker Stacks.towardsdatascience.comDeep Dive into Apache Spark DateTime FunctionsCatalog of DateTime functions in Apache Sparkmedium.comRunning Spark on Kubernetes: a fully functional example and why it makes sense for OLXSpark supports submitting jobs natively to a k8s scheduler since version 2.3.0, but even with its nice documentation…tech.olx.comHow to master Java transformations in AWS GlueAre you also enjoying Pyspark’s version av AWS Glue data transformations…engineering.klarna.comQuick start guide to Spark with DatabricksGetting results from Spark in minutes instead of daystowardsdatascience.comThings You Must Know About Spark RDDsIntroduction, Need and Features Of RDDsmedium.comApache FlinkReplayable Process Functions in Flink: Time, Ordering, and TimersHow Bird troubleshooted some tricky things involving using Flink+Kafka for robust data processing.medium.comCustom sources and sinks with FlinkBig data applications used to be, a long time ago, batches based on map-reduce. Some business domains, for instance…medium.comApache Flink for Continuous Insight Based on Event StreamsPractically all business-related data is produced as an infinite stream of events: sensor measurements, website…medium.comStreaming ETL With Apache Flink - DZone Big DataStreaming data computation is becoming more and more common with the growing Big Data landscape. Many enterprises are…dzone.comApache KafkaThe Surprisingly Simple Solution for Streaming DataKafka is a prime example of a project that uses a very basic immutable data structure — the log — to great effectmedium.comRunning Kafka on Azure Kubernetes ServiceYes, Kubernetes was not designed for Stateful workloads. Yes StatefulSets and PersistentVolumes can be hard to reason…medium.comDo not reinvent the wheel — use Kafka ConnectPractical usage of Kafka connectors and some basics needed to start working with them.blog.softwaremill.comKafdropAn Open-Source Kafka UItowardsdatascience.comKafka Stream Processing — Handling DeletionsWith stream-table duality, deletions are usually represented by events with a Null value. For an example, think of the…medium.comTutorial : Validate your data pipeline using Schema Registry and Kafka Streams.TLDR ;medium.comSelf-serve data unification with KafkaIt’s no secret in the oil and gas industry that trying to unify and make sense of data is a difficult problem. Data…medium.comNear Real-Time Synchronization with Kafka Connect and Kafka Streams — Part 1This is part one of two blog posts about how we built a near real-time synchronization mechanism for classified ad data…tech.willhaben.atKSQL and KStreams for monitoring event-driven pipelinesIn our last article we told the story about how to implement a scalable, event-driven based CRM archtcture. We also…medium.comApache BeamGetting a Graph Representation of a Pipeline in Apache BeamGetting a pipeline representation in Apache Beam explained step-by-step.medium.comAdvent of Code 2019 in Apache Beam (Days 1 and 2)Concepts: Pipeline, Map, Combine, Create, FlatMapmedium.comGetting Started to Build Pipelines with Apache BeamWhat is Apache Beam?medium.comApache HadoopWhat is Hadoop?Learn more about this big-data powerhousemedium.comHadoop Ecosystem: Hadoop Tools for Crunching Big Data - DZone Big DataIn this blog, let's understand the Hadoop Ecosystem. It is an essential topic to understand before you start working…dzone.comApache Hadoop: How MapReduce Can Essentiate Data From HDFS - DZone Big DataData generation is becoming an important process. As the world economic forum states, a total of 463 exabytes (10,006…dzone.comApache NiFiModern Apache NiFi Load Balancing - DZone Big DataIn today's Apache NiFi, there is a new and improved means of load balancing data between nodes in a cluster. With the…dzone.comApache AirflowImplementing the functional data engineering paradigm in data load processes by using AirflowWe integrate several data sources in a project comprising more than forty applications. Such sources are integrated by…towardsdatascience.comApache Airflow Tips and Best PracticesThe intermediate guide to building reliable data pipelines with Airflow.towardsdatascience.comBuild Data Pipelines with Apache AirflowThe beginner’s guide to Apache Airflow.towardsdatascience.comAWSAWS Data Lake And Amazon Athena Federated QueriesSupercharge your AWS data lake architecture with new query service extensionsblog.openbridge.comAn AWS Data Lake with S3 Explained!If you’ve had any connection to the data world, you’ve probably heard some memorable, often quirky phrase about how…towardsdatascience.comGetting Started with Apache Zeppelin on Amazon EMR, using AWS Glue, RDS, and S3Explore the use of Apache Zeppelin on Amazon EMR for data analytics and data science, using a series of pre-made…towardsdatascience.comHow to master Java transformations in AWS GlueAre you also enjoying Pyspark’s version av AWS Glue data transformations…engineering.klarna.comMaking sense of Stream Data with AWSThis post is a continuation of Making sense out of Stream data locally.medium.comSecure Data Lake with AWS Lake Formation TutorialAWS Lake Formation helps to build a secure data lake on data in AWS S3. This blog will help you get started by…medium.comAWS Glue — ETL Microsoft SQL Server to AWS Aurora — MySQLAWS Glue can be used to extract, transform and load the Microsoft SQL Server (MSSQL) database data into AWS Aurora —…medium.comAWS Lake Formation And Glue Access AnalyzerAWS Lake Formation permissions control access to data sets in your data lake in AWS at a table and column level…medium.comETL and ELT design patterns for lake house architecture using Amazon Redshift: Part 1 | Amazon Web…Part 1 of this multi-post series discusses design best practices for building scalable ETL (extract, transform, load)…aws.amazon.comUnder the hood: Scaling your Kinesis data streams | Amazon Web ServicesReal-time delivery of data and insights enables businesses to pivot quickly in response to changes in demand, user…aws.amazon.comGoogle CloudHistorical load into Ingestion Time partitioned BigQuery table using Dataflow1. Introductionmedium.comNoSQLSQL or NoSQL?NoSQL database is more and more popular in the modern data architecture. It has become a powerful way to store data in…towardsdatascience.comRealtime Data in Apache Druid — Choosing the Right StrategyWhy you should use Kafka Indexing instead of Tranquilitytowardsdatascience.comDashboard ToolsApache Superset — a cost-effective alternative to quick sight for data lakeA tale of gluing apache superset, AWS Athena, headless firefox, gecko driver and selenium to get the amazing experience…medium.comETLSQL vs noSQL: Two Approaches to ETL ApplicationsIn this article I will explore the differences in SQL and noSQL ETL pipelines.towardsdatascience.comELT Data Pipelining in Snowflake Data Warehouse — using Streams and TasksIn my previous post:medium.comMicrosoft ETL: Understanding ETL with Azure Data FactoryWith multiple options and configurations available for Microsoft, we’ll explain the use of ETL with Microsoft and Azure…medium.comBuilding an ETL process (or ELT really) with Step Functions and LambdasProblem Statement: Extract and transform all data in a fast and scalable manner.medium.com
