Stop Pending Steps From Failing Your CI Build

TDD is a big part of how I develop software. I use TDD to help me design the software I’m writing as well as serve as a to-do list for the things that need to be built for the app. That means that sometimes I write tests for things that I know I want to include but not at the moment. These are smaller steps of the bigger feature but possibly not the core functionality. For example, I might want to track some events and I want to keep a reminder of myself not to forget that step, but at the moment I want to focus on the main feature itself and then add the tracking part once I’ve got the basic stuff working.

To make that work, I write pending tests that will show that they are pending in the output. That said, I typically do not want a pending test to fail the test suite. Especially in a new app that has a bunch of core things that need to be built first, before getting to some of the fringe items like event tracking. This workflow works well in my opinion, except for one gotcha, which I will cover in this post.

If you are using Cucumber and CircleCI and running the suite using the typical command bundler exec cucumber you might see something like this at the end of the run:

10 scenarios (0 undefined, 1 pending, 9 passed)
120 steps (0 undefined, 1 pending, 119 passed)
0m17.414s

You can implement step definitions for undefined steps with these snippets:

Then("foo") do
  pending # Write code here that turns the phrase above into concrete actions
end

Coverage report generated for Cucumber Features, MiniTest to /home/circleci/project/coverage. 968 / 1064 LOC (90.98%) covered.
Exited with code 1

And while you do not have any failing tests, the build is marked as failing and in red. How come?

The last line gives us a hint. The fact that cucumber returned an exit code of 1, as pointed out in the Exited with code 1 line of the output of the run, is the reason why CircleCI has marked the build as failed. So if you’d like your build to pass even though you’ve got pending steps, you need to change the output from cucumber to be a 0 instead of 1.

Cucumber has the option --[no-]strict that you can use to make it exit with a code 0 instead of 1 for a bunch of scenarios. The flag is set to be on by default (i.e --strict) which means it returns a 1 and if you turn it off it would return a 0. The scenarios that are grouped under the general strict flag are undefined, pending, and flaky. I personally only want it to skip pending steps because those are purposefully left unimplemented. In that case I can use --no-string-pending as the flag. I learned about these flags from the PR: https://github.com/cucumber/cucumber-ruby/pull/1169.

Now that we are aware of the flag to use, we can just update the config for CircleCI and push the changes up and see if that works. But builds take a while to finish, and I wanted to make sure that this would work before pushing it up to CircleCI. One problem though: the terminal does not print out the exit codes of each command you run, that would be silly. So how can I make sure that the flag does what I want then? I remembered a RubyTapas episode in which Avdi talks about system variables like $1, $?, $! and others and how to use them in Ruby. I knew one of them contains the exit code of the previous process, I just needed to find out which one and then print it out in the terminal after executing the cucumber command with and without the flag and see if that made any difference. Turns out $? is the one.

So I ran the following:

ychaker ~/dev/app$ be cucumber
ychaker ~/dev/app$ echo $?
1
ychaker ~/dev/app$ be cucumber --no-strict-pending
ychaker ~/dev/app$ echo $?
0

which validates that adding the --no-strict-pending flag will help resolve the issue of marking the build as failed on CircleCI. After changing the CircleCI config to:

- run:
    name: Run cucumber tests
    command: bundle exec cucumber --no-strict-pending

the build started passing.

There you go. That’s the trick. Some times it’s important to know that something exists, or that there’s a tool out there that you can leverage in certain scenarios, even if you’re not an expert in it. As long as you know what to look for, you can then figure it out. Half the battle is knowing what you don’t know so that you can learn it.