Conducto for CI/CD
Execution Environment
In this tutorial, you will learn how to specify the dependencies and code necessary for your commands to run. Conducto strives to make this as simple as possible.
When we walked through creating your first pipeline, we glossed over an important detail — specifying the execution environment of your commands. That is, for each command, you must be able to specify:
Explore our live demo, view the source code for this tutorial, or clone the demo and run it for yourself.
git clone https://github.com/conducto/demo.git
cd demo/cicd
python execution_env.py --local
Alternatively, download the zip archive here.
Containers and Images
Conducto achieves this by running each of your exec node commands inside of a docker container, which is defined by an image that you help to configure. An image is a template for an execution environment that contains a base operating system and filesystem contents, including libraries, packages, and user code. A container is an instantiation of an image, and is like virtual machine, but lighter weight and quicker to create and destroy.
We will deep dive into how you configure an image. As a refresher, this is the pipeline from your first pipeline tutorial, with the image parameter bolded.
import conducto as codef build_and_test() -> co.Serial:
image = co.Image(image="golang:1.14", copy_dir="./code")
with co.Serial(image=image) as pipeline:
with co.Parallel(name="build"):
co.Exec("go build -x auth.go", name="auth")
co.Exec("go build -x app.go", name="app")
with co.Serial(name="test"):
co.Exec("go run auth.go --test", name="auth")
co.Exec("go run app.go --test", name="app")
return pipelineif __name__ == "__main__":
co.main(default=build_and_test)
Image Specification
In Conducto, there are two ways to specify an image.
- Specifying an existing image from DockerHub or another image registry.
- Specifying a custom Dockerfile.
Existing Image
Specifying a existing image looks like this.
image = co.Image("node:lts-buster")
This particular image contains Node.js in a Debian “buster” operating system, and is one of the many official Node.js images available on DockerHub. You can specify any image from any public image registry, or a locally built image.
Python Image + Python Requirements
If you specify an image with python installed, we also allow you to specify any python package requirements inline.
image = co.Image("python:3.8-alpine", reqs_py=["PTable"])
This specific example is equivalent to having python 3.8 installed in Alpine Linux, with the following pip command having been run.
pip install PTable
Custom Dockerfile
For more control, you can specify your own Dockerfile, which Conducto will build into an image. You may specify dockerfile
with an absolute or relative path, which is evaluated relative to the location of your pipeline script. You must also specify context
, which is the docker build context.
image = co.Image(
dockerfile="./docker/Dockerfile.simple",
context="."
)
Here is a very simple Dockerfile that results in an image equivalent to the python example from the previous section.
FROM python:3.8-alpine
RUN pip install PTable
Adding Your Own Code
So far we have discussed how to use images to include required software dependencies. But, you likely also need to include your own code in the image.
There are a few ways to do this.
- Copy a local directory directly into the image.
- Clone a specific branch from a git repository into the image.
- COPY or ADD files explicitly in a Dockerfile.
Copy a Local Directory
You can specify a local directory with your own files to be copied into your image with the copy_dir
argument. You may use an absolute or relative path for the directory, which is evaluated relative to the location of your pipeline script.
image = co.Image("python:3.8-alpine", copy_dir="./code")
This copies the directory ./code
into your image. You may specify copy_dir
for any version of image specification from above: existing image or dockerfile.
Clone from Git
You can also specify a git repository and branch to clone into your image with the copy_url
and copy_branch
arguments. This is especially useful for running a CI/CD pipeline against a git pull request. Here is an example using our demo repo on GitHub.
git_url = f"https://github.com/conducto/demo.git
image = co.Image(
"python:3.8-alpine", copy_url=git_url, copy_branch="master"
)
Just like copy_dir
, you can specify copy_url
and copy_branch
to any version of image specification.
COPY or ADD in Dockerfile
Finally, if you specify your own custom Dockerfile, you can COPY or ADD any files you want. Here is a Dockerfile that explicitly copies a code
directory into the image. In this example, ./code
is a path relative to the docker build context, specified by the context
argument as seen earlier.
FROM python:3.8-alpine
COPY ./code /root/code
Mounting Local Code for Debugging
One of our favorite features in Conducto is live debugging. We show an example of this in our debugging tutorial. When you debug a node, you get a shell in a container with your full execution environment, including any code you have added to the image. If possible, we will mount your local code, creating a live debug environment. In this mode, any edits you make to your code outside of the container are visible inside the container, where you can test your command in its full execution environment. This allows you to use your regular editor and debug tools outside of the container to make the debug process as painless as possible.
We can do this in two scenarios:
- you add your code with
copy_dir
, or - you specify
path_map
to explicitly map paths outside the container to inside the container.
So, you get the feature for no effort if you use copy_dir
, but you have to specify an extra parameter if you want to use live debug with the clone from git or dockerfile image specifications.
Clone from Git + path_map
If you always have a local checkout of the git repo that you specify to an image, you can safely specify a path_map
to make any later debugging easier. Here is the example from above with path_map
added.
git_url = f"https://github.com/conducto/demo.git
path_map = {".": "cicd"}
image = co.Image(
"python:3.8-alpine",
copy_url=git_url,
copy_branch="master",
path_map=path_map
)
This maps the local directory .
, relative to the location of the pipeline script, which is outside the container, to the cicd
directory relative to the root of the cloned git repo inside the container.
COPY or ADD in Dockerfile + path_map
It works the same way for a image with a dockerfile that adds its own files, except that the target path inside the container must be absolute. This is because in this scenario, Conducto has no way to choose a reasonable default root directory inside the container. Here is an example.
path_map = {"./code": "/root/code"}
image = co.Image(
dockerfile="./docker/Dockerfile.copy",
context=".",
path_map=path_map
)
Where the Dockerfile is the same as above.
FROM python:3.8-alpine
COPY ./code /root/code
Image Inheritance
Finally, a node with unspecified image
parameter will inherit the values of it’s parent. The pipeline from our first tutorial shows this, with all nodes sharing an image with the root node.
import conducto as codef build_and_test() -> co.Serial:
image = co.Image(image="golang:1.14", copy_dir="./code")
with co.Serial(image=image) as pipeline:
with co.Parallel(name="build"):
co.Exec("go build -x auth.go", name="auth")
co.Exec("go build -x app.go", name="app")
with co.Serial(name="test"):
co.Exec("go run auth.go --test", name="auth")
co.Exec("go run app.go --test", name="app")
return pipelineif __name__ == "__main__":
co.main(default=build_and_test)
That is all there is to it! Now, with the information you learned in Your First Pipeline, Environment Variables and Secrets, Data Stores, Node Parameters, CI/CD Extras, and here, you can create arbitrarily complex pipelines.