You can build pipelines out of commands in any language with Conducto, but we have some extra support for python that allows you to easily glue python functions together into rich and dynamic pipelines. Pass a python function to co.Exec. Lazily define your pipeline at runtime with co.Lazy. Use Markdown…

Anyone who has spent time with complex data science pipelines has spent a lot of that time resolving errors with them. Bugs are just a reality when you are trying to implement a complex system. …

Anyone who has spent time with complex CI/CD pipelines has spent a lot of that time resolving errors with them. Bugs are just a reality when you are trying to implement a complex system. …

Exec, Serial, and Parallel Nodes support several parameters that make pipeline specification in Conducto extremely powerful. You have already learned about image and env. You can also specify: cpu and mem to constrain resources requires_docker to run docker commands stop_on_error to implement the finally pattern same_container to control container sharing …

Data science pipelines necessarily generate data, plots, or intermediate results that need to be stored for some amount of time. You cannot simply persist these files on the local filesystem, because each command runs in a container with it’s own filesystem that disappears when the container exits. And, in cloud…

A pipeline is a sequence of commands that must be executed in a specific order. Some steps can happen concurrently, while other steps must happen one after another. Conducto is a tool for writing, executing, visualizing, and debugging pipelines. At its most basic level, Conducto makes it trivial to chain…

Non-trivial pipelines require the specification of environment variables and secrets. This is easy in Conducto. Explore our live demo, view the source code for this tutorial, or clone the demo and run it for yourself. git clone https://github.com/conducto/demo.git cd demo/data_science python env_secrets.py --local Alternatively, download the zip archive here. Environment Variables To…

In this tutorial, you will learn how to specify the dependencies and code necessary for your commands to run. Conducto strives to make this as simple as possible. When we walked through creating your first pipeline, we glossed over an important detail — specifying the execution environment of your commands…

In this tutorial, you will learn how to define, execute, and interact with a simple Conducto pipeline. Upon completion, you will understand how to use the following minimal API. co.Exec, co.Serial, and co.Parallel node classes, co.Image to specify execution environment, and co.main() to make your pipeline executable Explore our live…