Building a Flexible Powerful Alerting System in Orchestra
Introduction
We’re extremely excited to announce our new alerting feature within Orchestra.
Alerting is an incredibly important part of any Data Operation. However, it’s an area where a software engineering approach has limitations. How can logging extract helpful information from systems such as Snowflake? How can systems be unified using a single alerting service? How can Business Stakeholders be brought in?
These are all questions software engineering-style alerts can’t answer, which is why we’re happy to announce Orchestra is now capable of all these things (almost).
🚀 Want to try it now? You can here 🚀
Alerting to Anyone, Anywhere, Anytime
Within Orchestra, Analytics Teams build end-to-end data pipelines.
This doesn’t just include operations happening in your data warehouse (perhaps being orchestrated with dbt or Coalesce) but data ingestion tasks using services like Portable or Airbyte, jobs running on Cloud infrastructure like AWS ECS or EC2, and Dashboard refreshes in Tableau or Power BI 🔗.
By building Data and AI Products in Orchestra, fast-moving businesses also get end-to-end alerting out of the box. This is incredibly versatile and can save companies weeks if not months building custom alerting systems, significantly improves Trust in the Data Team, and can decrease time spent debugging by up to 30%. Read on more to find out here.
Multiple destinations
Orchestra supports Slack, Microsoft Teams and Email as destinations.
By providing a wrapper around these heterogeneous APIs / Services, Data Teams do not need to spend valuable time building custom-alerting systems and can instead focus on building Data and AI Pipelines, knowing that alerting with Orchestra just works.
Multiple conditions
Take advantage of Orchestra’s different completed states to bring in Business Stakeholders. You can send a different alert to a different place depending on what state a particular Task lands in. Here’s a powerful example:
- Data ingestion process failure → #data-engineering-failures
- Data ingestion process succeeded -> #data-engineering-business-as-usual
- Dashboard Refresh ends in skipped/failed -> #business-stakeholder-alerts
- Dashboard Refresh ends in succeeded -> #business-stakeholder-alerts
Here, we see with 4 configurations Data Engineers have everything they need to accurately monitor Data Pipelines without getting alert fatigue. The high degree of configurability ensures the amount of alerts sent is not too many but also not too few.
Importantly, business stakeholders also get alerted when their dashboards become stale or impacted by upstream process changes. This can all be handled from Orchestra, which is significantly less scary for Non-Technical users than the Airflow UI!
Multiple resources
Send alerts at a Pipeline or Task level. This is incredibly powerful as it means that if there is a specific owner for specific Tasks or specific parts of a Data Pipeline, they can be informed instantly and accurately.
Unlimited alerts with infinite scalability
There is no limit on the number of alerts that can be sent from Orchestra. Orchestra can be leveraged as a piece of serverless infrastructure that scales infinitely to send alerts and minimises the strain on platform engineering from data teams.
Key benefits
There are a number of key benefits to leveraging Orchestra for end-to-end alerting.
Increased trust
By ensuring 100% coverage of your data estate, users of Orchestra are always the first to know about issues in Data and AI Pipelines. This makes them more reliable and efficient than if alerts were configured in multiple different places
Increased Product and AI Velocity
Decreased Reliance on Platform Teams and time spent building and maintaining custom alerting systems frees up time to build Data and AI Products.
Almost 50% of Data Teams reported building their own data alerting system at some point, despite the fact this is boilerplate work. Leveraging Orchestra stops teams from wasting valuable time maintaining and scaling home-grown alert systems.
Ease of Use
Rather than require a code change that gets lost in the pile of PRs, anyone can set up an alert instantly in Orchestra using the UI. If this needs to be git-controlled, it will soon be able to be via Orchestra’s Git integration.
This makes the process of managing alerts significantly faster and easier to manage.
Conclusion
There is still much to do to become the perfect alerting platform for Data Teams.
While Fast-growing businesses using Orchestra get complete end-to-end coverage, in organisations of enormous scale there are inevitably trickier problems that call for more elegant solutions.
These include but are not limited to:
- Great customisation in alerts
- Leverage Orchestra as a “Reactive” system of alerting (i.e. Orchestra receives logging information and sends alerts)
- dbt-native alerting for Analytics Engineers
We’re excited to see what you think of our v 1.0 of alerting and can’t wait to hear your feedback.
Find out more about Orchestra
Orchestra is a low-code data orchestration and data observability platform. It’s also a build any DAG you can build in Airflow using Orchestra. You can use it to solve many use cases and solutions like swiftly building data products, preserving data quality, and even data governance. Our docs are here, but why not also check out our integrations — we manage these so you can get started with your pipelines instantly. We also have a blog, written by the Orchestra team + guest writers, and some whitepapers for more in-depth reads.
