Testing Bash applications

Nikita Sobolev
Jul 8, 2017 · 6 min read

Some time ago I was faced with a task of testing a bash script. At first I decided to use Python unit-tests, however, I was reluctant to bring external technologies to the project. Therefore I had to go with the testing framework written in the notorious bash.

Overview of the existing solutions

After googling available solutions, I was presented with very scarce options. We are going have a closer look at some of them.

Which criteria are going to be important?

  1. Dependencies: when taking a bash testing framework you wouldn’t want it to drag python, lua, and a few more systems packages along with it

assert.sh

One of the first options that I noticed was a small framework assert.sh. It is a pretty good solution — easy to install and use. In order to write the first test, you need to create a file tests.sh (example taken from the documentation):

. assert.sh

# `echo test` is expected to write "test" on stdout
assert "echo test" "test"
# `seq 3` is expected to print "1", "2" and "3" on different lines
assert "seq 3" "1\n2\n3"
# exit code of `true` is expected to be 0
assert_raises "true"
# exit code of `false` is expected to be 1
assert_raises "false" 1
# end of test suite
assert_end examples

Then you can run this file:

$ ./tests.sh
all 4 examples tests passed in 0.014s.

Other advantages include:

  • Simple syntax and use

However, there is a number of serious drawbacks:

  • At the time of writing the article, there was a red icon on, saying “build failing” on Github, this looks scary

Conclusion: it is a good tool, which I would recommend using if you need to write a simple tests for a basic shell script. It isn’t suitable for more complex tasks.

shunit2

Installing shunit2 is not as easy as the previous tool. I was unable to find an adequate repository — there is some project on Google Code, there are a few on Github, left at various stages 3 and 5 years ago, and there are even some svn repositories. Consequently, it is impossible to make sense which release is the latest and how to download it. But those are small inconveniences.

How do the tests themselves look? Here is a simplified example from the documentation:

testAdding()
{
result=`expr 1 + 2`
assertEquals \
"the result of '${result}' was wrong" \
3 "${result}"
}

And then running it:

/bin/bash math_test.sh testAdding  Ran 1 test.  OK

This framework boasts of some unique features for its class:

  • Ability to create test suites inside the code. This feature can be handy when you have tests for certain platforms or shells. In this case, you can use your own namespaces, such as zsh_, debian_, etc

Still, there are a few considerable disadvantages, which have pushed me away in the end:

  • The project is almost inactive

Conclusion: it is a serious tool, which should be setup in a flexible enough way and make an indispensable part of your project. However, lack of structure in the shunit2 project itself is scary, so I decided to continue my search.

roundup

Initially, I was intrigued by this framework because it was written by the author of Sinatra for ruby. I also liked test syntax, which resembles the well-known mocha. All functions starting with it_ inside the file are considered as tests and run by default. Interestingly, all tests run in their own sandbox, which allows avoiding extra errors. Here is how an example from the documentation looks:

describe "roundup(5)"before() {
foo="bar"
}
after() {
rm -f foo.txt
}
it_runs_before() {
test "$foo" "=" "bar"
}

There are no examples of the output. You need to install it and check by yourself, which is not that good, actually. On the plus side, though:

  • Each test runs in its own sandbox, which is very convenient

Still, there are quite a few drawbacks:

  • You cannot create a source of common functions for all the tests (To be completely honest you can if you use a hack)

Conclusion: it is a perfectly mediocre tool, you can not say it’s a good one, and yet it isn’t that bad. In its functions, while being wider, are similar to assert.sh. When should you use it? If you were going for assert.sh, and the only things lacking are functions before() or after().

bats

I say it straight away — I chose this framework in the end. There is a lot to like. First of all, great documentation: examples of use, semantic versioning; and I would like to specifically point out the list of projects using bats.

bats uses the following approach: the test is considered complete if all the commands inside return code 0 (like set –e does). Here is how test written on bats look like:

#!/usr/bin/env bats@test "addition using bc" {
result="$(echo 2+2 | bc)"
[ "$result" -eq 4 ]
}
@test "addition using dc" {
result="$(echo 2 2+p | dc)"
[ "$result" -eq 4 ]
}

And the output:

$ bats addition.bats
✓ addition using bc
✓ addition using dc
2 tests, 0 failures

You can get test information in text compatible with Test Anything Protocol with the help of a flag --tap. You can find the plugins for a wide number of programs there: Jenkins, Redmine, SublimeText and others.

Apart from peculiar test syntax, there are other interesting things about bats:

  • Command run allows you to run the command first and then test its outgoing code and text output; which you can do with the help of special variables: $status and $output

I have already listed quite a large number of positive bats’ features. As for the negative sides, I was able to find only one:

  • bats steps away from valid bash. Tests should be written in files with .bats extension, using a different shebang

Conclusion: it is a quality tool with close to no weaknesses. I highly recommend it.

Results

This research was made in attempt to write some quality tests for my personal project called git-secret. Which primary goal is to store encrypted files in the git repository. Check it out:

wemake.services

We love Python and Elixir. We use Javascript.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store