🚀 Houston: A better approach to workflow orchestration

Complex workflows (data pipelines, ML prediction pipelines, optimisation algorithms, etc.) always need orchestration.

Software like Airflow orchestrates and runs tasks within a single ‘monolithic’ service. The server becomes slow, unresponsive, and expensive as it grows.

We propose a new approach that orchestrates independent microservices remotely via a simple API. Our solution is now available as the free open-source software Houston.

A simple workflow, represented as a graph (Houston UI)

The orchestration API server is light-weight and easy to deploy:

Starting a Houston API instance via the command line. Docker and Terraform deployments are just as easy

Defining new services (task runners) is dead simple. We’ve hidden all the complexity. For example, here’s a Google Cloud Function, written in Python, that’s fully integrated with Houston:

from houston.gcp.cloud_function import service

@service()
def main(file_path):
   file = load(file_path)
   upload_file(file)

5 key advantages:

• Most (if not all) code runs on serverless tools, which lowers cost, maintenance, and co-dependencies vs a managed option¹
• New workflow runs start up instantly and the UI loads really fast because the orchestration server has relatively little work to do → one API instance can easily handle 500+ concurrent workflows²
• Multiple workflows can use the same services at the same time, allowing code and infrastructure to be shared across teams more easily
• A workflow can be made up of microservices written in various languages and runtimes, running on different cloud providers, allowing teams to make gradual upgrades as new technology becomes available with no vendor lock-in
• Optionally, a publisher/subscriber message system is used to trigger each stage to run, meaning message delivery is guaranteed and services don’t get overloaded by too many requests; they pull messages at their own pace

Disadvantages:

• All code, no matter how simple, must be deployed as a microservice. This increases the development overhead. We’ve countered this by creating utilities to speed up the creation of microservices
• Workflow errors can go undetected if there is insufficient monitoring
• State cannot be persisted between stages without an external database

How it works

A collection of isolated microservices, working together to complete a task, fits quite well with the metaphor of a space mission:

• Microservices are the rockets, landers, and satellites. They’re completely isolated from each other
• The API is the mission control centre, which orchestrates everything. It tells the services what to do, and which services should run next.
• Not everything goes to plan… but we can find ways to recover from any failures and keep the mission going

This is why we called the orchestration software Houston.

“Okay, Houston…we’ve had a problem here” — Jim Lovell, Apollo-13 Mission

The mission timeline view from the Houston UI

Why we made it

We’re a Data Science consultancy with many different clients. We needed isolated instances for every client and environment. Airflow would be too expensive (imagine 3 environments per client per project, running heavy duty ML workloads, costing £500 a month each!).

Equally important is the developer experience. Houston is designed to be simple and fast. Entire workflows can finish in under a second³ thanks to a lightning fast server written in Go. The dashboard UI is always live, showing updates as they happen, unlike Airflow’s frustratingly slow dashboard.

Ease of maintenance is also a huge concern for us. We share the same microservices across many projects, allowing us to upgrade and test them only once and be completely confident that they’ll run healthily in every project. We also avoid the need for huge migrations with extended downtime (e.g. upgrading a large Airflow cluster from Python 3.11 to 3.12).

We also hate how different tools don’t talk to each other. Why do we have separate pipelines for Dataflow, Data Factory, Vertex AI, etc. all in one project? Houston is completely platform agnostic, so we can use it on any cloud provider and any tool.

…

It’s easy to get started, so try it out today!

👉 callhouston.io

👉 github.com/datasparq-ai/houston

1. Deloitte Consulting. ‘Determining the Total Cost of Ownership of Serverless Technologies when compared to Traditional Cloud’. September 2019. https://pages.awscloud.com/rs/112-TZM-766/images/AWS_MAD_Deloitte_TCO_paper.pdf

2. See a live demonstration of this by running houston demo --stress-test

3. See the local server quickstart for an example of this. A plan with 6 stages completes almost instantly when the stages do nothing because the API calls between the service and the server are in the order of a few milliseconds.