Data Engineering Digest #6 (November 2019)Maycon Viana Bordin·FollowPublished indata.plumbers·7 min read·Dec 10, 2019--ListenSharePhoto by Pixabay from PexelsNew ToolsOn dataset versioning in Palantir FoundryHow we drive collaborative data engineering through dataset versioning and sandboxingmedium.comIntroducing Azkarra Streams : The first micro-framework for Apache Kafka StreamsAzkarra Streams is a new open-source micro Java framework that let you focus on writing Kafka Streams topologies code…medium.comData Engineering RoleOne Year As A Data EngineerI’ve just recently past the one year mark of working as a Data Engineer at a fin-tech company. I’ve learnt loads in…blog.usejournal.comCourses & TrainingTop 10 Courses to learn Big Data and Hadoop — Best of LotIf you are learning Big Data, or, want to explore the Hadoop framework, and are looking for some awesome courses, then…medium.comHow to become a Hadoop Developer? — Job Trends and SalaryThis article is a comprehensive guide on how to become a Hadoop Developer, its job trends and salary trends.medium.comHow to become Google Cloud certified Professional Data Engineer (updated exam 2019)?If you have been thinking about taking the exam but don’t know whether your preparation is good enough to pass the exam…medium.comNewsDatabricks Ranked as one of North America's Fastest Growing Companies for the Second Consecutive…SAN FRANCISCO--()-- Databricks, the leader in unified data analytics, today announced it has been ranked as one of…www.businesswire.comData CultureThe Three Levels of Data Analysis- A Framework for Assessing Data Organization MaturityThis blog post is Unfiltered If I had a nickel for every time I saw that Data Science Hierarchy of Needs visual in a…about.gitlab.comHow to Create a Successful Data StrategyWith rapid advances in AI and data science, data has become an essential asset to every enterprise. Setting up a data…towardsdatascience.comData Engineering NuggetsAn introduction to the disorganized tire fire that is my data-driven mindtowardsdatascience.comData LakeData Lake vs Data Warehouse: Do You Need Both? - DZone Big DataMost enterprises today have a data warehouse in place that is accessed by a variety of BI tools to aid in the…dzone.comWhy going real time?We have a data lake, now what ?medium.comFrom Data Lakes to OceansAggregating into data lakes is the solution of today — but are Federated Sources the solution of tomorrow?medium.comData Quality & GovernanceBig Data Quality at CriteoWhat is Data Quality and why it matters more than you think on Hadoop.medium.comSearching Flipkart Data Catalog with Apache JenaWe use Apache Jena extensively to create a data catalog system of our data platform. We have extended it to allow…tech.flipkart.comBoosting the Data Governance journey with Google Cloud Data CatalogThoughts on data discovery and metadata management in Google Cloudtowardsdatascience.comHealthcare Analytics & Data GovernanceBy Bonnie Chung, Vanna Trieu and Matthew Trisicmedium.comImplementing Data Quality with Amazon Deequ & Apache SparkData quality is an important aspect of big data ingestion. Here, we’ll discuss how we can implement that using Amazon…medium.comStorageBig Data File Formats Explained - DZone Big DataIn this article, we discuss the positives and negatives of using several common big data file formats, including CSVs…dzone.comCompressing Your Big Data: Tips and Tricks - DZone Big DataThe growth of big data has created a demand for ever-increasing processing power and efficient storage. DigitalGlobe's…dzone.comDemystify Hadoop Data Formats: Avro, ORC and ParquetCore concepts and use cases of three data formats widely used in Hadoop: Avro, ORC, and Parquet.towardsdatascience.comThe New Metrics of Object Storage - The New StackAs a general rule, when people think about object storage they think about one thing - the price per TB/GB. Though a…thenewstack.ioStream ProcessingCreating a Real-Time Flight-info Data Pipeline with StreamSets, Kafka, ElasticSearch and KibanaIn this article, I will use the Schiphol Flight API, StreamSets Data Collector, Apache Kafka, ElastichSearch and Kibana…itnext.ioAnalytics on Kafka Event Streams Using Druid, Elasticsearch, and Rockset - DZone Big DataEverything you need to get started analyzing Kafka Event Streams Events are messages that are sent by a system to…dzone.comData VirtualizationData Virtualization - The Evolution of the Data Lake - DZone Big DataThe idea of the traditional data center being centered on relational database technology is quickly evolving. The…dzone.comData PipelinesDeploying and versioning data pipelines at scaleTom Goldenberg, Data Engineer, Musa Bilal, Machine Learning Engineer, QuantumBlackmedium.comAntelopes for pulling Data CartsReal-time customer journeys using Kudumedium.comBatch Processing Pipelines for Better Data AnalysisHow we generate intelligible insights from our data warehouse using batch pipelines.blog.gojekengineering.comSparkApache Spark-3.0 Sneek peakApache Spark has remained strong over the years and now is coming back with one of its major releases with its ongoing…towardsdatascience.comGlimpse into Spark 3.0 [Early Access]Tushar Kapoor: (https://www.tusharck.com/)towardsdatascience.comIntroduction to Delta Lake : Part 1A tutorial on how to use Delta lake with Apache Spark.medium.comApache Spark Unit Testing Part 2 — Spark SQLSecond part of article series about how to use Spark repository classes for Unit Testing. Spark SQL package has four…medium.comFlinkConsuming Kafka Messages From Apache Flink - DZone IntegrationConsuming Kafka Messages From Apache Flink In my previous post, I introduced a simple Apache Flink example, which just…dzone.comStreamr integration templates to Apache FlinkIs Apache Flink a better real-time streaming data processing engine than Spark? Yes. And that’s why we integrated it…medium.comHadoopWhat went far wrong with big data?Baggage of historical choices and JVM/GC. Let’s learn lessons from Telco industry.medium.comFeed file Generation from data stored in HDFS and exposed using HiveProblem Statement:medium.comIntroduction to HadoopWhat is Hadoop: Architecture, Modules, Advantages, Historytowardsdatascience.comBig Data From B to A: The Hadoop Distributed Filesystem — HDFSYour guide to understand HDFS conceptstowardsdatascience.comSolving stability problems in Hadoop cluster — big data with small dataExperiencing stability problems with Hadoop cluster and the key concepts to solve such problems.medium.comKafkaSingle Message Transforms in Kafka ConnectChris Matta Print Kafka Connect is the part of Apache Kafka ® that provides reliable, scalable, distributed streaming…www.confluent.ioHow to install Kafka using DockerSet up a Kafka cluster for test and development purposeitnext.ioKafka Administration and Monitoring UI Tools - DZone Big DataKafka itself comes with command line tools that can perform all necessary administrative tasks. But, those tools aren't…dzone.comProcessing guarantees in KafkaEach of the projects I’ve worked on in the last few years has involved a distributed message system such as AWS SQS…medium.comDeploy a Kafka Cluster with Terraform and AnsibleA simple way to provision, deploy and manage a Kafka platformtowardsdatascience.comKafka InternalsThis article gives a glimpse on what exactly happens when a message is produced from Kafka , how it is stored in broker…medium.comIntroduction to Event Streaming with Kafka and KafdropMaking sense of modern architectural trends in data and application engineeringmedium.comKafka GotchasGreat, but not Perfectitnext.ioApache PulsarIntroduction to Apache Pulsar — Concepts, Architecture & Java ClientsApache Pulsar is an open-source distributed streaming platform which was originally created at Yahoo.medium.comAirflowOrganize your Big Data Pipeline using AirflowIf you’re struggling to streamline and organize your workflow in a big data pipeline, airflow might be the magic bullet…medium.comHow to Use Airflow without HeadachesTutorial about how you can implement Airflow Tasks and DAGs without problems and with joy.towardsdatascience.comUsing Airflow and Spark To Crunch US Immigration DataDefining the end-result and code interfaces to simplify a sticky data engineering task.towardsdatascience.comAirbnb’s Airflow Versus Spotify’s LuigiGetting data from point A to point Bmedium.comBringing Machine Learning models into production without effort at DailymotionHow we manage to schedule Machine Learning pipelines seamlessly with Airflow and Kubernetesmedium.comScaling Apache Airflow for Machine Learning WorkflowsLearn how to easily execute Airflow tasks on the cloud and get automatic version control for each machine learning…towardsdatascience.comKeeping your ML model in shape with Kafka, Airflow and MLFlowHow to incrementally update your ML model in an automated way as new training data becomes availablemedium.comRunning Airflow at KabbageWhat is Airflowmedium.comTesting in Airflow Part 2 — Integration Tests and End-To-End Pipeline TestsWritten by Sarang Shinde and Chandu Kavar — Nov 28, 2019medium.comAirflow On Azure Kubernetes — Part 1How to deploy Airflow on AKS using Helmmedium.comKey Airflow Pipeline Deployment Lessons On KubernetesAt Natural Intelligence we use Airflow extensively in order to create simple ETL flows as well as complex ML tasks that…medium.comAirflow: Externally trigger a DAG when a condition matchOverall, I need to build a data pipeline architecture pulling data from operational databases and import them into our…medium.comNiFiExploring Apache NiFi 1.10: Parameters and Stateless Engine - DZone Big Datahttps://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12344993 You can now use JDK 8 or JDK…dzone.comAWSAWS Glue: Why Should Enterprises Use It? - DZone CloudUse AWS Glue as your ETL tool of choice. AWS Glue is a fully managed extract, transform, and load (ETL) service that…dzone.comBuilding a great ML platform using Jupyter Hub, SageMaker and spark on AWS.A scalable, powerful, interactive and multiuser ready platform for analytics and machine learning, 100% cloud.medium.comStream Kinesis data to spark streaming[AWS EMR]Recently, we did a project on streaming data from Kinesis to spark streaming.We used spark streaming as we needed to do…medium.comOptimize Spark jobs on EMR ClusterThis article is for Data Scientists and ML Engineers who are running Spark jobs using EMR cluster on AWS and have faced…medium.comQuick tips on tweaking spark configuration and performance on EMRTweaking Spark configuration for beginner might be a daunting task as the required knowledge ranges from understanding…medium.comSpark at Onera: A Hybrid Cloud ApproachThe Best of EMR and Dataprocmedium.comNoSQLMongoDB Tutorials and Articles: The Complete Collection - DZone DatabaseEverything you need to get started with MongoDB In this edition of "Best of DZone," we take a look at one of the most…dzone.comTileDB: A Database for Data ScientistsWhy Data Science needs a new DBmedium.comLords of ACIDby Andrew C. Olivermedium.comRedis cluster performance on a single machine vs KeyDB multithreadingA few interesting points taken from this article: https://docs.keydb.dev/blog/2019/10/28/blog-post/medium.comTuning DynamoDB: Scans vs QueriesAs we continue our digital transformation to support our streaming platfor m— Redbox On Demand — we are finding new…medium.comCAP Theorem Considered Harmful: Data integrity in NoSQL & Alternatives to CAP“CAP is best avoided”. So states the acclaimed “Designing Data-Intensive Applications”, a best seller on distributed…medium.com
