Data Engineering

The COVID-19 dataset Global.health has great promise, but is badly flawed. Its data is old and low quality. — Summary The COVID-19 dataset Global.health (G.h) began almost three years ago with great promise. It is funded by Google.org, Rockefeller Foundation, Boston Children’s Hospital and others, with academic support from University of Oxford, Northeastern University, Harvard Medical School and others. Its salient feature is individual person-level data for COVID-19 cases worldwide…

Leverage ChatGPT to become more efficient — Social media is abuzz with chatters about how ChatGPT can generate recipes for exotic cuisines, write fictional and non-fictional books, recommend travel itineraries, create marketing collateral and blogs, suggest content for posting videos, write code for various machine learning tasks, and what not. However, you can extract even more from…

Ensuring that Apache Spark is tuned for optimal performance involves adjusting various parameters based on your workload, cluster resources, and use case. Here are some important parameters to consider when tuning Apache Spark: 1.Spark executor configuration: spark.executor.memory: The amount of memory allocated per executor. Adjust this value based on available…

Looking at a data engineer job description, it is easy to fall into the trap of thinking that it’s all about building data pipelines. But this is just the tip of the iceberg of all the duties of a data engineer. In this article, I will highlight the three other…

A conceptual representation of data and its relationships is created through the process of data modelling. It entails specifying the data’s structure in a way that makes information processing, storage, and retrieval efficient. In this post, we’ll explore the various kinds of data modelling, their applications, and how data models…

Step by step implementation of CI/CD for your data pipelines in 20 mins. Contents The Data Lifecycle What is CI/CD for data? Why do you need CI/CD for data? CI/CD for data with lakeFS hooks Data quality guard rails with lakeFS pre-merge hook Implementing CI/CD with lakeFS and Docker (in 10…

This is going to be a really short article outlining solutions to running concurrent dbt schedules and their limitations. Solution to multiple schedules So you want to run some models on two different schedules because running everything every 15 minutes is expensive. The solution is simple; you specify tags for your models per below. …

Note: This article is part three of a series of guides designed to help data engineers who are new to Python. You can find part one (packages, modules, __init__.py, pip and imports) and part two (dependency management and virtual environments) on the Dagster blog. Python is a versatile and widely-adopted…