Note: This article is part three of a series of guides designed to help data engineers who are new to Python. You can find part one (packages, modules, __init__.py, pip and imports) and part two (dependency management and virtual environments) on the Dagster blog. Python is a versatile and widely-adopted…

Shalabh Chaturvedi Serverless development and feedback loops With Serverless Dagster Cloud you can develop and deploy Dagster code without setting up either a local development environment or any cloud infrastructure. When you commit a change to GitHub, a GitHub Action builds and deploys your code directly to Dagster Cloud, where you can view and interact…

March 16, 2023 * 12 minute read * In this blog post, we explore the topic of fake GitHub stars. We will share our approach for identifying them and invite you to run this analysis on repos you are interested in. Click here to skip the background story and jump…

By Sandy Ryza | March 20, 2023 If you build or operate data pipelines, there's a good chance you think a lot about, or should think a lot about, partitions. Partitioning is a technique that helps data engineers and ML engineers organize data and the computations that produce that data…

Dagster runs on Python, and most data engineers or developers with a basic grasp of Python can get simple pipelines up and running rapidly. But some users who are less familiar with Python find Python packages to be a bit of a headache. So let's talk about what Python packages…

The newly released `dagster-airflow` library has made migrating off legacy Airflow and onto Dagster much easier.

by Pete Hunt, CEO of Elementl, inc. ChatGPT, ever heard of it? ChatGPT came out a few months ago and blew everyone’s minds with its ability to answer questions sourced from a broad set of knowledge. Around the time that ChatGPT was demonstrating how powerful large language models could be…

In this post we’re going to talk about the process of moving from an ETL script to a robust Dagster pipeline using Software-Defined Assets. 🧠 What is ETL? ETL is an acronym for “Extract-Transform-Load.” This is one of…

by Sandy Ryza New files arrive on S3. Within five minutes, the core tables that depend on them are rebuilt. In time for the daily 9 AM executive meeting, the daily rollup table is recomputed from those core tables. You push a bug fix to the code that derives one…