Conducto for CI/CD

Rapid and Painless Debugging

Rapid and painless are rarely words associated with debugging real-world CI/CD pipelines. But with Conducto, they are a reality. When we switched our internal CI/CD pipeline from CircleCI to Conducto, our typical debug cycle dropped from 30+ minutes to less than 3 minutes. This is not an exaggeration.

This is because Conducto empowers you to:

• quickly identify errors in the web app,
• fix specification errors in-place,
• immediately reproduce an error in a shell,
• debug locally with your own editor and tools, and
• effortlessly push fixed code into a live pipeline.

To demonstrate, I will debug some real errors in Conducto’s internal CI/CD pipeline.

Check out our related Easy Error Resolution tutorial here.

Quickly Identify Errors

I have made some changes in the Conducto code base, and I want to run our CI/CD pipeline to make sure they are ok. I launch the pipeline, and soon after, I see an error in the left hand pipeline pane.

The pipeline summary is the row at the top, the pipeline pane is on the left, and the node pane is on the right. The pipeline pane shows execution status, including one error so far.

Here is a closer look at the pipeline pane. We see node execution state: Pending, Queued, Running, Done, Errored, and Killed. One node is in the Errored state.

I quickly jump to the error by clicking the Next Error icon.

The Next Error icon is boxed in yellow.

It turns out that we know app_test is a flaky, and we just have not gotten around to writing a robust fix for it yet. For now, I know that my changes have nothing to do with this part of the code, so I choose to skip the test. I select the app_test node and click the Skip button, which removes that node from the pipeline, allowing the pipeline to continue.

The Skip icon is boxed in yellow. I have skipped the app_test and all of its subnodes. We no longer have any nodes in the Errored state.

Fix Specification Errors In-place

My pipeline continues to run, and I finally encounter an real error related to my changes. I click the Next Error icon to jump to it.

This error is definitely my fault. Time to debug.

The node pane on the right shows me detailed information, including the command it ran, execution parameters, and stdout and stderr if present.

The node pane shows me useful information like the command and stderr. I can see the error right away.

Luckily, this is an easy bug. I removed the --tag argument from build.py and simply forgot to update the command in the pipeline script. I can quickly make an in-place modification to the command to fix this pipeline.

Select the Modify icon in the pipeline pane. It is boxed in yellow.

Delete the tag argument from the command.

After clicking UPDATE, I see the change I have made in the node pane.

I can confirm that the only change I made was to delete the tag argument.

Now, I simply click Reset in the pipeline pane to re-run the node and see it succeed.

The Reset icon is boxed in yellow. Clicking it re-runs the manager node, which passes this time.

Note that this fix is local to this instance of the pipeline, and does not modify anything in our pipeline script. So, at this point, I also put the fix in my pipeline script so that future pipelines do not suffer from this error.

Immediately Reproduce an Error in a Shell

Finally, my pipeline fails on the run node, our final integration test.

One more hurdle before a successful run.

The node pane stderr shows me that this is a code error in the file test_integration.py. I actually need to debug this problem, so I click on the Live Debug lightning bug icon.

I actually need to debug this error. I click the Live Debug icon, boxed in yellow.

This copies a command like this to my clipboard.

conducto livedebug --id=ehb-xna --node=/integration/pipelines/run --timestamp=1586384556.384000

I can paste this into any shell with conducto installed and drop into a container with my command and full execution environment.

The first box shows the live debug command. Running it drops me into a docker container, where I have the command that I need to debug.

As suggested in the instructions printed by livedebug, I can execute this command and see the exact stderr reported in the web app.

This is the same error we see in the web app. This means that our debug environment works.

Debug Locally With Your Own Editor and Tools

This container has mounted my local code, which means that I can use my editor of choice (VSCode, PyCharm, and vim are all popular choices at Conducto) and any other debug tools on this problem. Once I fix the code, the livedebug container automatically sees my edits, and I can simply re-run the command to confirm that it works.

The livedebug container automatically sees my fixes, and now the test passes.

Effortlessly Push Fixed Code into a Live Pipeline

The final step is to push my fixed code into the pipeline and re-run any affected nodes. This is incredibly easy, I just click the Rebuild and Reset button in the node pane.

The Rebuild and Reset button is boxed in yellow in the upper right. The Timeline is boxed in yellow below it.

Rebuild and Reset re-builds the image with your latest code, then resets the node so that it runs again. The Timeline shows the previous failed run, and the new successful one. You can click on any run in the Timeline to see it’s command, execution parameters, stdout, and stderr to keep track of what has changed from run to run.

Why Are You Still Waiting?

And that is our typical test-debug-test cycle at Conducto. In prior jobs, I have used Jenkins and CircleCI, and I spent a lot of time painstakingly re-creating debug environments and waiting around when testing a fix required re-running an entire pipeline. Now, I test frequently and debug quickly and effortlessly. Simply put, using Conducto for CI/CD makes me more productive, by a lot.

Why are you still waiting? Get started with Conducto now. Local mode is always free and is only limited by the cpu and memory on your machine. Cloud mode gives you immediate scale. Use the full power of python to write pipelines with ease. And, experience rapid and painless debugging.