Testing command line tool output with RSpec

The RSpec Relish app is one of my favorite documentation sites, but I didn’t find out about testing output there, I was researching static analysers for ruby and looking in their spec folders to see what approach they took. I can’t remember which gem it was but I discovered that RSpec already had my back.

Here is the basic approach:

expect { system %(echo “gotta love rspec”) }
 .to output(a_string_including(“gotta love rspec”))
 .to_stdout_from_any_process

If you are not familiar with the system call, it will create a new process and execute the string you passed to it — imagine opening up another terminal window and typing in the string and pressing enter, it’s pretty much doing that. I am wrapping the argument to system with %() to remove the hassle of managing quotes.

You can also use system to run an executable in your path, let's create a basic one to demonstrate:

#!/usr/bin/env ruby
puts “look at me run”

Save that file as runner and then run chmod +x runner on it so that we can execute it just by typing ./runner and pressing enter on the command line. To write a test spec for this executable we can use system to execute it in another process:

expect { system %(./runner)) }
 .to output(a_string_including(“look at me run”))
 .to_stdout_from_any_process

So you see these two examples are pretty much the same, but it is easier to demonstrate without having to create new executable files left, right and centre. Tally ho!

Matching Strings

To not have to worry about matching the exact output, whitespace and carriage returns included, I am using a_string_including, so as long as this string exists in the output then it will pass.

Without a_string_including we get this:

expect { system %(echo “hello command line world”) }
 .to output(“hello command line world”)
 .to_stdout_from_any_process
# => expected block to output “hello command line world” to stdout, but output “hello command line world\n”

Notice the carriage return at the end \n, let's just avoid those shall we.

Where does it go?

Lastly, I’m using the to_stdout_from_any_process to specify where I am expecting the output to be sent to, you must choose where the output should go after using output or you will get an RSpec error:

RuntimeError:
 You must chain `to_stdout` or `to_stderr` off of the `output(…)` matcher.

The to_stdout method will not work in this use case because our call to system is echoing the message to a child process. To use to_stdout you would need to be testing what is printed to STDOUT in this running process:

expect { puts “from the same process” }.to output(“from the same process”).to_stdout

Which is useful for when you are writing assertions against a programmatic entry point into the tool:

module CLI
 def self.call
   puts “Thanks for stopping by”
 end
end
RSpec.describe CLI do
 specify do
   expect { CLI.call }.to output(
     a_string_including(“Thanks for stopping by”)
   ).to_stdout
 end
end

Most of your testing will be done at this level, so to_stdout is what you need, but to make sure running the executable is working you will need to match against the child processes with from_any_process.

Testing Errors

Did you notice in the error message from RSpec that the other suggestion is to use to_stderr? If you didn’t know or couldn’t guess this is where output that is regarded as an error goes.

You won’t generally use this for unit testing, but when invoking your command-line tool any exceptions that are raised by Ruby will go to STDERR. For instance, a file named crash (that we have run chmod +x on):

#!/usr/bin/env ruby
NoConfig = Class.new(StandardError)
raise NoConfig, “no configuration”

Could be tested like:

RSpec.describe “crash” do
  subject { system “./crash” }
  it { expect { subject }.to output.to_stderr_from_any_process }
  specify “must be configured” do
    expect { system “./crash” }
      .to output(a_string_including(“no configuration”))
      .to_stderr_from_any_process
  end
end

Specifying that a certain string is output is optional as you can see in the first expectation.

Conclusion

This is how I have been using the output matcher when building the tollgate gem with TDD.

Have a good read through the RSpec relish docs for the output matcher for more information on its usage.

Gotta love RSpec :)