Vertex AI Pipelines vs. Cloud Composer for ML Orchestration

A personal opinion based on workshops with many different customers

Back in the days, at a time when ML was not all over the place.
Companies already did a lot of data processing for other purposes. A large number of Cloud Composer workflows got created.

Nowadays companies often try to implement machine learning workflows into their existing Cloud Composer workflows. While this might work, you should at least consider using Vertex AI Pipelines.

This article is for you if you came to the conclusion that your previously only data-related workflows are migrating more and more into an ML workflow.

I want to highlight there is no right or wrong you might use both or just one of the products.

Enough blah blah let's get to my personal opinion about it.

• Both are simple in usage, a task or component can be defined as simple python code or as a container.
• Passing data between tasks or components is similar, large data requires the use of intermediate storage.
• The overall usage of setting up the environment, and getting an understanding of how to actually implement a workflow/pipeline is easier with Vertex…