Data Engineering Weekly #4
The story is a cross-posting from the Data Engineering Weekly. Please subscribe to the Data Engineering Newsletter for the latest update.
Welcome to the fourth edition of the data engineering newsletter. This week’s release is a new set of articles that focus on data orchestration, ML applications, tuning data workload, and Kafka on Kubernetes.
Airflow is a huge step forward over loosely coupled cron jobs for running the data pipeline. Dagster, a data-aware, typed, self-describing, logical orchestration graph, takes the data orchestration to the next level by focusing on local development, testable code before production, and Linking data assets to the code that produced them. The focus on data dependencies, not with pure execution dependencies, is a data engineer’s dream comes true.
Amundsen is a data discovery and metadata engine, open-sourced by Lyft joining LF AI Foundation.
Vimeo writes a post on video social analytics infrastructure using Apache Spark. The major challenge around integrating the external API guarded with severe rate limits. The practical usage of micro batching to workaround external API rate limiting and decouple the application logic from API data sourcing is a pragmatic approach and an exciting read.
Slack writes about ML infrastructure to prevent spam invites. The key takeaway is the simplicity of the approach and focuses on the operational aspect of the ML application.
Koalas is an open-source project which provides a drop-in replacement for pandas that focuses on scalability. Databricks writes a post on how PySpark can effectively work with Koalas.
EMR is the widely used big data service from AWS. Monitoring Amazon EMR clusters is essential to help detect critical issues with the applications or infrastructure in real-time and identify root causes quickly. AWS writes about how to integrate EMR metrics with Prometheus and other monitoring ecosystems such as Grafana for dashboarding and AWS SNS to send notifications and alerts.
The Buy vs. Build on the table when it comes to stream processing considering the complexity of the system. Apache Kafka and AWS Kinesis are the leading competitors when it comes to message brokers. It’s (not) surprising that Apache Kafka still years ahead in stream processing.
Strimzi is an open-source CNCF sandbox project that focuses on running Apache Kafka on Kubernetes while providing container images for Apache Kafka itself, Zookeeper, and other components that are part of the Strimzi ecosystem. The blog post narrates how to move the Apache Kafka workload to Kubernetes.
Apache Kafka consumers are a single-threaded processing model that follows one partition consumed per thread. The model simplifies the ordering and processing guarantee in processing the stream of events. The downside of the approach, we often underutilize the CPU. Confluent writes a blog post that narrates how can we implement Multi-Threaded Message Consumption with the Apache Kafka Consumer and the challenges around it.
Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent the opinions of current, former, or future employers.
