Conducto for CI/CD
Supercharge your CI/CD with Conducto
I have used and heavily relied upon several CI/CD tools throughout my career: Jenkins, CircleCI, make, and home-grown scripts. They were all fine up to a point, and then they weren’t. I have never met a developer intimately familiar with any of these or similar tools who loves them. So, I was extremely surprised when the data science pipeline tool we built, unintentionally turned out to be really, really good for CI/CD.
At Conducto, we initially used CircleCI for our internal CI/CD. As our code base and infrastructure grew in complexity, CircleCI became increasingly difficult to use. When we switched to Conducto, we made sure that the tool addressed all of the pain points we had been experiencing.
Specification
YAML is the de-facto choice for configuration in most other CI/CD tools. It is fine for limited use cases, but for anything non-trivial it becomes a pale imitation of a complete programming language. When we ported our CI/CD pipeline to Conducto, the specification went from an ~1800 line YAML file to a few python scripts with 40% fewer lines. Most importantly writing our Conducto pipeline in python was significantly faster, less error-prone, and easier to debug than our original brittle YAML config file and its tangled web of dependencies.
Visualization
Every other CI/CD tool represents a pipeline as a static DAG (Directed Acyclic Graph), which is basically a workflow with no loops. DAGS are theoretically ideal, but we have yet to see one that shows a useful summary of a pipeline with more than 10–20 nodes. Our pipeline with 70+ unit tests, infrastructure deployment steps, and integration tests resulted in a mess of blurry spaghetti. It was either so zoomed in it had no context, or so zoomed out that any text was illegible.
Conducto’s interactive tree representation evokes a file browser, something every developer is familiar with. Tree views are intuitive and dense with information, allowing a user to start with a macro view and dig down to see details. They expose as much or as little detail as desired, in a compact space. Pipeline trees easily surface execution states and errors, enabling rapid test-debug-test cycles.
Execution
Other CI/CD tools run in the cloud, for a price. They offer limited free access that is not suitable for anything beyond a simple trial. Debugging is also in the cloud, and limited to the tools that can be installed in a shell. Pushing a fix requires re-running an entire pipeline, with added cost and delay.
Conducto’s container-based architecture enables many powerful features. Local mode is always free and is only limited by the CPU and memory on a machine. The same pipeline runs in cloud mode for effortless scalability. Debugging occurs on a local machine using typical development environment and debugging tools. Fixes can be deployed to live pipelines, by restarting whatever subset of the pipeline is relevant, without wasting time re-running parts of the pipeline that are unaffected.
Why Are You Still Waiting?
Why are you still waiting for your builds to finish in your current tool? Get started with Conducto now and immediately increase your productivity.
