Conducto for CI/CD

Your First CI/CD Pipeline

In this tutorial, you will learn how to define, execute, and interact with a simple Conducto pipeline.

Upon completion, you will understand how to use the following minimal API.

• co.Exec, co.Serial, and co.Parallel node classes,
• co.Image to specify execution environment, and
• co.main() to make your pipeline executable

Explore our live demo, view the source code for this tutorial, or clone the demo and run it for yourself.

git clone https://github.com/conducto/demo.git
cd demo/cicd
python first_pipeline.py --local

Alternatively, download the zip archive here.

Define Your Pipeline

In Conducto, you express your pipeline as a series of commands that need to be executed in serial and/or parallel. Our python API exposes a minimal set of Node classes to get this done quickly and painlessly. Then, you have the full power of python to nest these nodes for arbitrarily complex pipelines.

First, you need to import conducto.

import conducto as co

Then, you start building your pipeline with nodes.

Exec Node

An exec node simply wraps a shell command.

auth_test = co.Exec("go run auth.go --test")

Serial Node

A serial node specifies that a series of sub-nodes must happen in one after another. If one of the sub-nodes fails, execution stops and the entire serial node is marked as failed.

test = co.Serial()
test["auth"] = co.Exec("go run auth.go --test")
test["app"]  = co.Exec("go run app.go --test")

Parallel Node

A parallel node specifies that a series of sub-nodes can occur in parallel. All nodes are executed, and if any nodes fail, the entire parallel node is marked as failed.

build = co.Parallel()
build["auth"] = co.Exec("go build -x auth.go")
build["app"] = co.Exec("go build -x app.go")

Nesting

Serial and parallel nodes may contain any node type, not just exec nodes. This allows the creation of non-trivial pipelines.

pipeline = co.Serial()
pipeline["build"] = co.Parallel()
pipeline["build"]["auth"] = co.Exec("go build -x auth.go")
pipeline["build"]["app"] = co.Exec("go build -x app.go")
pipeline["test"] = co.Serial()
pipeline["test"]["auth"] = co.Exec("go run auth.go --test")
pipeline["test"]["app"]  = co.Exec("go run app.go --test")

Easy to do, but perhaps more verbose than you prefer. We can use python to make it nicer.

with co.Serial() as pipeline:
    with co.Parallel(name="build"):
        co.Exec("go build -x auth.go", name="auth")
        co.Exec("go build -x app.go", name="app")
    with co.Serial(name="test"):
        co.Exec("go run auth.go --test", name="auth")
        co.Exec("go run app.go --test", name="app")

Image

Of course, your commands will only be able to run in an execution environment with:

• your software dependencies installed,
• a copy of your own code present, and
• any necessary environment variables set

Conducto achieves this by running each of your exec commands inside of a docker container, which is defined by an image that you help to configure. Read full details in the Execution Environment and Environment Variables and Secrets tutorials. But for now, we will skip over these details, and just provide an appropriate image for our example. This particular image includes software to build and run go files, and copies over your local ./code directory. Note that the . is relative to the location of the pipeline script.

image = co.Image(image="golang:1.14", copy_dir="./code")
with co.Serial(image=image) as pipeline:
    # ...

Main

Now that you have a pipeline specified, make it executable. First, wrap your pipeline in a function that returns the top-level node.

def build_and_test() -> co.Serial:
    image = co.Image(image="golang:1.14", copy_dir="./code")
    with co.Serial(image=image) as pipeline:
        with co.Parallel(name="build"):
            co.Exec("go build -x auth.go", name="auth")
        # ...
    return pipeline

Conducto requires that you write a type hint to indicate the node return type of the function. Do not worry if type hints are new to you. Simply ensure that the first line of your function includes -> co.[NodeClass], like this:

def build_and_test() -> co.Serial:

Finally, define the main function of your python script.

def build_and_test() -> co.Serial:
    image = co.Image(image="golang:1.14", copy_dir="./code")
    with co.Serial(image=image) as pipeline:
        with co.Parallel(name="build"):
            co.Exec("go build -x auth.go", name="data_app")
        # ...
    return pipeline
if __name__ == "__main__":
    co.main(default=build_and_test)

Execute Your Pipeline

Executing your pipeline is easy. First, if you want to spot-check your pipeline, run your script with no arguments.

python first_pipeline.py

You will see a pipeline serialization like this.

/
├─0 build
│ ├─ auth   go build -x auth.go
│ └─ app   go build -x app.go
└─1 test
  ├─0 auth   go run auth.go --test
  └─1 app   go run app.go --test

To execute the pipeline on your local machine, which is always free, run this. Note that in local mode, your code never leaves your machine.

python first_pipeline.py --local

Coming soon, you will be able to effortlessly run the same pipeline in the cloud too.

python first_pipeline.py --cloud

Interact With Your Pipeline

The script will print a URL and pop it open in your browser. You can view your pipeline,

The pipeline summary is the row at the top, the pipeline pane is on the left, and the node pane is on the right. The pipeline pane shows your pipeline, with parallel, serial, and exec nodes getting unique icons.

run it and quickly identify pipeline status,

Press the Run button in the upper left of the pipeline pane. See the execution status of each node: Pending, Queued, Running, Done, Errored, and Killed.

examine the output of any exec node,

View the command, execution params, stdout, and stderr of a node in the right hand node pane.

and rapidly and painlessly debug errors. Collaborate with anyone else in your org by sharing the URL.

Put your pipeline to sleep when you are finished with it. Its state, logs, and data are stored for 7 days. During this period you can wake it up. After 7 days, it is deleted.

The “zzz” icon in the pipeline summary puts the pipeline to sleep. When no pipelines are selected you see a list of available ones. Click the “alarm clock” button on a sleeping pipeline to get a wakeup command to run into a local shell.

Why Are You Still Waiting?

This was a simple example, but once you add in execution environment, environment variables and secrets, data stores, node parameters, and CI/CD extras, you can easily express the most complex of CI/CD pipelines in Conducto.

In prior jobs, I have used CircleCI and Jenkins, spending way too much time wrangling brittle YAML files and inefficiently clicking through configuration screens. Now, my pipelines are only limited by what I can express in python, which is both simple and powerful. Simply put, using Conducto for CI/CD makes me more productive, by a lot.

Why are you still waiting? Get started with Conducto now. Local mode is always free and is only limited by the cpu and memory on your machine. Cloud mode gives you immediate scale. Use the full power of python to write pipelines with ease. And, experience rapid and painless debugging.