The Prefect Blog
Published in

The Prefect Blog

Modular Data Stack — Build a Data Platform with Prefect, dbt and Snowflake (Part 5)

Failure handling with alert notifications and retries for data ingestion and dbt transformations in your Modern Data Stack

Marvin observing, orchestrating, and coordinating your dataflow universe and helping you build a reliable data platform through observability, alerting, and retries
Table of contents· Failure handling: alert notifications & retries
Alerting
🤖 Configure Slack alerts on failure
🤖 Failing successfully with automated retries
🤖 Trigger a flow run with a task that will fail and retry
📚 Summary of alerts & automated retries
· 🤖 Manual retries from the UI
🤖 Important notes ❗️
🤖 Trigger a run from deployment to test alerts
🤖 Retry the flow run manually
🤖 Inspect the logs
📚 Summary of data ingestion workflows
· Simple dbt transformation flow
Parametrization
🤖 Jaffle shop: dbt build from the Prefect UI
🤖 Inspecting the dbt run from the UI
🤖 Alerts on failure in dbt tests
📚 Summary of simple dbt transformation flows
· Parsing the dbt manifest
Pros and cons of parsing the dbt manifest
Using the Dbt Prefect block
🤖 Orchestrating dbt from manifest
🤖 View Slack alert and troubleshoot with the Prefect UI
🤖 Fixing that issue with manual retries
📚 Summary of flows parsing the dbt manifest
· Next steps

Failure handling: alert notifications & retries

Alerting

🤖 Configure Slack alerts on failure

🤖 Failing successfully with automated retries

🤖 Trigger a flow run with a task that will fail and retry

python flows/ingestion/ingest_jaffle_shop_randomly_failing.py
The flow run has successfully completed upon a retry

📚 Summary of alerts & automated retries

🤖 Manual retries from the UI

🤖 Important notes ❗️

prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT=true

🤖 Trigger a run from deployment to test alerts

prefect deployment run raw-data-jaffle-shop/local-process

🤖 Retry the flow run manually

🤖 Inspect the logs

📚 Summary of data ingestion workflows

Simple dbt transformation flow

Parametrization

dbt build --select result:error+ --defer --state ./target
dbt run --select failed_model_name+

🤖 Jaffle shop: dbt build from the Prefect UI

🤖 Inspecting the dbt run from the UI

🤖 Alerts on failure in dbt tests

📚 Summary of simple dbt transformation flows

Parsing the dbt manifest

Pros and cons of parsing the dbt manifest

Failure alert saying which dbt model or test failed and providing a URL to the logs to investigate deeper

Using the Dbt Prefect block

🤖 Orchestrating dbt from manifest

🤖 From Slack alert to the logs in the Prefect UI

🤖 Fixing that issue with manual retries

📚 Summary of flows parsing the dbt manifest

Next steps

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store