How to Run Tagged Scala Tests with SBT and ScalaTest
--
We use SBT to build all of our Scala projects, and we’re still scratching the
surface of what the tool can do. One thing we needed to do recently was to
separate a module’s tests (built on scalatest) into groups. This is possible
using scalatest’s tagging capabilities, which can be trigged with a custom SBT
configuration. Although the documentation is out there, we couldn’t find a
complete example. This is a step-by-step guide.
So why is this useful? Here’s an illustrative example. We use
[vagrant][vagrant] to easily spin up local infrastructure in a known state.
Using vagrant we can test an application’s use of systems such as
[RabbitMQ][rmq], [Riak][riak], [Redis][redis], [MongoDB][mongo] etc. We build
tests that use this infrastructure so we know when we break things, and by
using actual running instances, we avoid the need for mocks.
This is effective, but we don’t necessarily want to run these tests on every
integration, especially on a loaded build server. We use the test tagging
technique to mark tests that require vagrant in an ad-hoc manner, and
configure our build system to exclude them based on run tasks. We
can use a separate, less frequently used task, to run them. This technique
could easily be extended for other categories of tests (e.g. slow,
requiring DB access etc). The bottom line is, it shows us when a test
failure is a genuine coding error, and keeps our confidence in CI results high.
There are various stages to follow. In this worked example we are
creating the `com.example.RequiresVagrant` tag to designate
tests that depend on external vagrant hosts.
# Tag your Tests
## Create a Tag
Tags are descendents of `org.scalatest.Tag`. in our example we need to add the
following object to `src/test/scala/com/example/project`.
[cc lang=”scala”]
package com.example.project
import org.scalatest.Tag
object RequiresVagrant extends Tag(“com.example.RequiresVagrant”)
[/cc]
## Tag the tests
The tests are tagged at the level of individual tests, not suites. That means
we are looking for `it` blocks. We simply need to add our tag to them as
follows:
[cc lang=”scala”]
package com.example.project
import com.example.RequiresVagrant
// …
class StorageSpec extends FunSpec with GivenWhenThen with BeforeAndAfterAll {
describe(“A Broker”) {
it(“Should correctly forward valid requests to the storage actor”,
RequiresVagrant) {
// …
}
}
}
[/cc]
This will ensure that, when `RequiresVagrant` is a filtered
tag, this test won’t be run.
## Make fixtures safe
Filtering tests will prevent attempts to use pieces of infrastructure that
aren’t there. However if there are fixtures or other pieces of setup code, we
may still attempt to contact that infrastructure. To prevent this we need to
take all that code and put it in a block that will only run when a test runs.
If we want our fixtures to be set up once per test, we can use a function
definition (i.e. the block will be run 0 to many times). If we want a value
that’s shared among tests, we can put the block in a lazy val (i.e. the block
will run 0 or 1 times).
[Scalatest documentation][st-doc] recommends an anonymous inner object.
So, for example, if a test required some sort of storage infrastructure, it
would need to be amended as below. Here we are using two [akka][akka] actors: one that
uses some logic to filter the messages it receives, and the other that
receives messages and stores them. Our tests verify that the messages we
expect are subsequently found in storage.
Original:
[cc lang=”scala”]
package com.example.project
import com.example.RequiresVagrant
// …
class StorageSpec extends FunSpec with GivenWhenThen with BeforeAndAfterAll {
implicit val sys = ActorSystem(“storageTests”)
val storageActor = sys.actorOf(Props[StorageActor])
val broker = sys.actorOf(Props(new StorageBroker(storageActor)))
describe(“A Broker”) {
it(“Should correctly forward valid requests to the storage actor”,
RequiresVagrant) {
// Use broker
// Test storage has happened
}
// other tests
}
}
[/cc]
Changed:
[cc lang=”scala”]
package com.example.project
import com.example.RequiresVagrant
class StorageSpec extends FunSpec with GivenWhenThen with BeforeAndAfterAll {
implicit val sys = ActorSystem(“storageTests”)
lazy val fixture = new {
val storageActor = sys.actorOf(Props[StorageActor])
val broker = sys.actorOf(Props(new StorageBroker(storageActor)))
}
describe(“A Broker”) {
import fixture._
it(“Should correctly forward valid items to the storage actor”,
RequiresVagrant) {
// use broker as before
// test storage as before
}
// other
}
}
[/cc]
Whereas the first example will skip the test but still try to instantiate the
storage subsystem, in the second example the fixture-using code is never run,
so the lazy val is never initialized. Note the `import` line. Although this
pulls in `fixture`’s namespace, and ensures most test code is unaffected
by our change, it won’t cause the object to be instantiated until use. Were
the test not excluded, the test actors would be used, and the enclosing object
would be instantiated the first time that happened. If you are using the
`beforeAll` method to do some initialization, you may also safely use the
fixture. `beforeAll` will only be called if some tests are due to run.
## Clean up safely
There may be more code using these fixtures that would be run even if
individual tests aren’t run. The afterAll method is an example. There are two
things you can do about these:
- If running the code but ignoring errors won’t slow you down, just wrap it in a
`try` and ignore caught exceptions.
- If there are timeouts etc that would delay you if the code is run, use a
`var cleanUp = false` flag. You can and set it in every test.
Then in your `afterAll` function, simply test the `cleanUp` flag.
## Test your tests
You should now be in a position to toggle the test on or off from the command
line. ScalaTest provides an `-l` command line switch to exclude certain tags.
SBT, in turn, provides a facility to pass command line arguments to the
underlying test system. Anything after ` — ` will be passed through.
Let’s try it out:
[cc lang=”scala”]
sbt “test-only com.example.project.BrokerSpec — -l com.example.RequiresVagrant
[/cc]
If your test concludes rapidly, without failures, you’re where you want to be.
# Configure SBT
As shown previously, it’s possible to send arguments to the underlying test
library with SBT’s `test-only` task. However, it’s not possible do this for
the `test` task. And, in any case, we don’t want to constantly provide
command-line arguments for a frequently invoked command. `test` and
`test-only` are built-in tasks provided by SBT’s `test` scope. It would be
good to copy this scope into a new configuration, say `local`, and add our
command-line argument to those tasks, to take effect when our scope is used.
SBT allows you
to do just that. It requires the following steps:
## Add a new configuration
First we ensure our SBT configuration (in `project/*Build.scala`) is
importing `sbt._` and `Keys._`, to give us access to the `Test`
built-in. We find our Project declaration and add a new configuration with:
[cc lang=”scala”]
lazy val kernel = Project(/*existing configuration*/).configs(LocalTest)
[/cc]
We define `LocalTest` config as
[cc lang=”scala”]
lazy val kernel = Project(/*existing configuration*/).configs(LocalTest)
lazy val LocalTest = config(“local”) extend(Test)
[/cc]
which tells SBT that we want to give the configuration a scope called `local` and expect to use the same task keys as the built-in test scope.
We then pull the regular test defaults in with:
[cc lang=”scala”]
lazy val kernel = Project( /*existing configuration*/).configs(LocalTest)
.settings(inConfig(LocalTest)(Defaults.testTasks): _*)
lazy val LocalTest = config(“local”) extend(Test)
[/cc]
Finally, we define our custom task configuration:
[cc lang=”scala”]
lazy val kernel = Project( /*existing configuration*/).configs(LocalTest)
.settings(inConfig(LocalTest)(Defaults.testTasks): _*)
.settings(testOptions in LocalTest := Seq(Tests.Argument(“-l”,
“com.example.RequiresVagrant”)))
lazy val LocalTest = config(“local”) extend(Test)
[/cc]
That says, when you run `local:test` or `local:test-only`, add the tag excluding arguments we used earlier.
## Test SBT
Check that your new configuration will exclude the tests you want to exclude:
[cc]
sbt local:test
[/cc]
It should be fairly clear if this has been successful.
Check you didn’t break those excluded tests by running everything:
[cc]
sbt test
[/cc]
## Configure your build server
Thanks to all our previous work, this is pretty easy. Simply replace whatever arcane incantation you were using in your build job with `local:test`, e.g.:
[cc]
sbt local:test package
[/cc]
And you’re done!
## Further reading
* [Scalatest documentation](http://www.scalatest.org/user_guide/tagging_your_tests) on tagging (covers FlatSpecs)
* [Scalatest documentation](http://www.scalatest.org/user_guide/using_the_runner#filtering) on filtering CL arguments
* [Scalatest documentation][st-doc] on FunSpecs (covers shared fixtures and tagging funspecs)
* [SBT testing capabilities](http://www.scala-sbt.org/release/docs/Detailed-Topics/Testing)
* [How to add command-line arguments to SBTs existing test tasks](http://un-jon.github.com/scala/2012/06/25/how-to-filter-tests-with-scalatest-and-sbt/)
* [Some more options for running specific tests from SBT](http://jackcoughonsoftware.blogspot.ca/2010/02/sbt-and-test-arguments.html)
[vagrant]: http://vagrantup.com
[rmq]: http://rabbitmq.com
[riak]: http://basho.com/riak
[redis]: http://redis.io
[mongo]: http://mongodb.org
[scalatest]: http://scalatest.org
[st-doc]: http://doc.scalatest.org/1.8/org/scalatest/FunSpec.html
[akka]: http://akka.io
