Conducto for CI/CD

CI/CD Extras

These are some nice extras for CI/CD that did not fit into other demos, but we wanted to share them in case they help.

• Pass a python function to an Exec node.
• Put Markdown in stdout/stderr.
• Check out our internal CI/CD configuration.

Explore our live demo, view the source code for this tutorial, or clone the demo and run it for yourself.

git clone https://github.com/conducto/demo.git
cd demo/cicd
python extras.py --local

Alternatively, download the zip archive here.

Pass a Python Function to an Exec Node

You can pass a python function and its arguments to an Exec node instead of a shell command. This comes with a few caveats:

• The function name cannot start with an underscore (_).
• The image must include the file with the function.
• The image must install conducto.
• You must set typical node parameters like image, env, doc, etc. outside of the constructor, either in a parent node or by setting the fields directly.

Here is an simple example function and an Exec node that calls it.

def no_types(name, thing):
    print(f"Hello {name}, goodnight {thing}.")
image_name = "python:3.8-alpine"
image = co.Image(image_name, copy_dir=".", reqs_py=["conducto"])
output = co.Exec(no_types, "world", thing="moon")
output.image = image

In the above example, it is assumed that both arguments are strings. But, if the function annotates in arguments with type hints or provides default values, then arguments with the basic types str, bytes, int, float, and bool are allowed. Here is an example.

def simple_types(price: float, count=3, show_sum=True):
    total = 0
    for i in range(count):
        total += price
        if show_sum:
            print(f"Item #{i}: ${price}. Total: {total}")
        else:
            print(f"Item #{i}: ${price}.")
image_name = "python:3.8-alpine"
image = co.Image(image_name, copy_dir=".", reqs_py=["conducto"])
output = co.Exec(simple_types, 3.5, count=4, show_sum=True)
output.image = image

And finally, Conducto also understands lists of basic types and some complex types like datetime.date. For custom types, define an object with a to_str method and a from_str static method. Here is a more complex example.

import datetime, typing
def complex_types(prices: typing.List[float], date: datetime.date):
    tomorrow = date + datetime.timedelta(days=1)
    print(f"Today is {date}. Tomorrow is {tomorrow}.")
    total = 0
    for price in prices:
        total += price
        print(f"This item costs {price}, total so far: {total}")
image_name = "python:3.8-alpine"
image = co.Image(image_name, copy_dir=".", reqs_py=["conducto"])
today = datetime.datetime.now().date()
output = co.Exec(complex_types, [3.5, 7.6, 10.0], date=today)
output.image = image

Markdown in Stdout

You can use Markdown in the stdout and stderr of any command. Just wrap any part of the output with:

<ConductoMarkdown>...</ConductoMarkdown>

This is especially useful if you need to show something like a plot, image, or table. Here is portion of a python script that has generates a plot, and then displays it with Markdown. In this case, we put the file we generate into conducto-temp-data (using the python interface), which conveniently gives us a URL for referencing it in the Markdown.

import conducto as co
# Some code to generate a plot...
# Save plot to file.
filename = "/tmp/plot.png"
fig.savefig(filename)
# Put file in tempdata to get a url for it.
co.temp_data.put(name="demo_plot", file=filename)
url = co.temp_data.url(name="demo_plot")
# Generate Markdown and reference plot by url.
markdown = f"""<ConductoMarkdown>
One _very_ useful thing I can do is show a **plot**:
![plot]({url} "a neat plot")
</ConductoMarkdown>"""
print(markdown)

And this is what it generates in the node pane for an Exec node that executes this script.

Check Out our Internal CI/CD Configuration

We think our CI/CD needs are probably fairly typical for anyone with a monorepo, so here we share our approach. Our CI/CD scenario is:

• We have a GitHub repo with all of our code, including CI/CD pipeline scripts.
• We always launch our CI/CD pipeline from a local checkout of this repo.
• We want our images to always include a full checkout of the repo.
• We want to run our CI/CD pipeline in two modes, dev mode and pr mode.
• In both casees, we want livedebug to work for easier debugging.

In dev mode we want to copy our local checkout of the repo into each image, including any local unstaged and uncommitted changes.

In pr mode (pull request mode) we want to git clone a given branch directly from github into the image.

The logic for this is encapsulated in the ImageFactory class in our demo, and you can view the source here.

Finally, as a convenience, we specify a single Dockerfile that installs the most common libraries and tools that we need for running and debugging our CI/CD pipeline, and use that for almost all of our nodes. You can view our Dockerfile file here.

Here is an example that demonstrates ImageFactory and our Dockerfile being used for both dev mode and pr mode pipelines. Using the ImageFactory allows configuring these two pipelines to be almost identical.

Dev Mode

ImageFactory.init()
dockerfile = "/path/to/Dockerfile"
dev_image = ImageFactory.get(
    dockerfile=dockerfile, reqs_py=["conducto"]
)
with co.Serial(image=dev_image, name="dev_mode"):
    co.Exec("ls", name="ls_code_from_local")
    # whatever other nodes you need

PR Mode

ImageFactory.init(branch="master")
dockerfile = "/path/to/Dockerfile"
pr_image = ImageFactory.get(
    dockerfile=dockerfile, reqs_py=["conducto"]
)
with co.Serial(image=pr_image, name="pr_mode"):
    co.Exec("ls", name="ls_code_from_git_clone")
    # whatever other nodes you need

That’s it! By now you should know how to construct some powerful CI/CD pipelines with Conducto. If you think you missed anything, check out our recommended reading list here.