Testing Bash applications

Nikita Sobolev
Jul 8, 2017 · 6 min read
Image for post
Image for post

Some time ago I was faced with a task of testing a bash script. At first I decided to use Python unit-tests, however, I was reluctant to bring external technologies to the project. Therefore I had to go with the testing framework written in the notorious bash.

Overview of the existing solutions

After googling available solutions, I was presented with very scarce options. We are going have a closer look at some of them.

Which criteria are going to be important?

assert.sh

One of the first options that I noticed was a small framework assert.sh. It is a pretty good solution — easy to install and use. In order to write the first test, you need to create a file tests.sh (example taken from the documentation):

. assert.sh

# `echo test` is expected to write "test" on stdout
assert "echo test" "test"
# `seq 3` is expected to print "1", "2" and "3" on different lines
assert "seq 3" "1\n2\n3"
# exit code of `true` is expected to be 0
assert_raises "true"
# exit code of `false` is expected to be 1
assert_raises "false" 1
# end of test suite
assert_end examples

Then you can run this file:

$ ./tests.sh
all 4 examples tests passed in 0.014s.

Other advantages include:

However, there is a number of serious drawbacks:

Conclusion: it is a good tool, which I would recommend using if you need to write a simple tests for a basic shell script. It isn’t suitable for more complex tasks.

shunit2

Installing shunit2 is not as easy as the previous tool. I was unable to find an adequate repository — there is some project on Google Code, there are a few on Github, left at various stages 3 and 5 years ago, and there are even some svn repositories. Consequently, it is impossible to make sense which release is the latest and how to download it. But those are small inconveniences.

How do the tests themselves look? Here is a simplified example from the documentation:

testAdding()
{
result=`expr 1 + 2`
assertEquals \
"the result of '${result}' was wrong" \
3 "${result}"
}

And then running it:

/bin/bash math_test.sh testAdding  Ran 1 test.  OK

This framework boasts of some unique features for its class:

Still, there are a few considerable disadvantages, which have pushed me away in the end:

Conclusion: it is a serious tool, which should be setup in a flexible enough way and make an indispensable part of your project. However, lack of structure in the shunit2 project itself is scary, so I decided to continue my search.

roundup

Initially, I was intrigued by this framework because it was written by the author of Sinatra for ruby. I also liked test syntax, which resembles the well-known mocha. All functions starting with it_ inside the file are considered as tests and run by default. Interestingly, all tests run in their own sandbox, which allows avoiding extra errors. Here is how an example from the documentation looks:

describe "roundup(5)"before() {
foo="bar"
}
after() {
rm -f foo.txt
}
it_runs_before() {
test "$foo" "=" "bar"
}

There are no examples of the output. You need to install it and check by yourself, which is not that good, actually. On the plus side, though:

Still, there are quite a few drawbacks:

Conclusion: it is a perfectly mediocre tool, you can not say it’s a good one, and yet it isn’t that bad. In its functions, while being wider, are similar to assert.sh. When should you use it? If you were going for assert.sh, and the only things lacking are functions before() or after().

bats

I say it straight away — I chose this framework in the end. There is a lot to like. First of all, great documentation: examples of use, semantic versioning; and I would like to specifically point out the list of projects using bats.

bats uses the following approach: the test is considered complete if all the commands inside return code 0 (like set –e does). Here is how test written on bats look like:

#!/usr/bin/env bats@test "addition using bc" {
result="$(echo 2+2 | bc)"
[ "$result" -eq 4 ]
}
@test "addition using dc" {
result="$(echo 2 2+p | dc)"
[ "$result" -eq 4 ]
}

And the output:

$ bats addition.bats
✓ addition using bc
✓ addition using dc
2 tests, 0 failures

You can get test information in text compatible with Test Anything Protocol with the help of a flag --tap. You can find the plugins for a wide number of programs there: Jenkins, Redmine, SublimeText and others.

Apart from peculiar test syntax, there are other interesting things about bats:

I have already listed quite a large number of positive bats’ features. As for the negative sides, I was able to find only one:

Conclusion: it is a quality tool with close to no weaknesses. I highly recommend it.

Results

This research was made in attempt to write some quality tests for my personal project called git-secret. Which primary goal is to store encrypted files in the git repository. Check it out:

wemake.services

We love Python and Elixir. We use Javascript.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store