AirSpec: Writing Tests As Plain Functions In Scala

AirSpec is a new testing framework for Scala and Scala.js.

Airframe: airspec

wvlet/airframe

AirSpec uses plain Scala functions for writing test cases. This style requires no extra learning cost if you already know Scala. For advanced users, dependency injection to test cases and property-based testing are supported optionally.

AirSpec has nice properties for writing tests in Scala:

• Simple usage: import wvlet.airspec._, then extends AirSpec trait.
• Using plain Scala classes and methods to define tests. Public methods in a class extending AirSpec trait will be your test cases.
• No annotation (like ones in JUnit5) aris necessary.
• Testing with simple assertions: assert(cond) , x shouldBe y, etc. No need to learn complex DSLs.
• Lifecycle management with Airframe DI: The arguments of test methods can be used to inject necessary services for running your tests. And also the lifecycle (e.g., start and shutdown) of the injected services can be managed by Airframe DI.
• Nesting and reusing test cases with context.run(spec)
• Handy keyword search for sbt: > testOnly -- (a pattern for class or method names)
• Property-based testing integrated with ScalaCheck
• Scala 2.11, 2.12, 2.13, and Scala.js support

We are now planning to add some missing features (e.g., better reporting, power assertions):

• Milestone: AirSpec 19

Motivation

In Scala, there are several rich testing frameworks like ScalaTests, Specs2, uTest, etc. We also have a simple testing framework like minitest. In 2019, Scala community has started an experiment to creating a nano-testing framework (nanotest-strawman) based on minitest so that Scala users can have some standards for writing tests in Scala without introducing third-party dependencies.

A problem here is, in order to write tests in Scala, we usually have only two choices: learning DSLs or being a minimalist.

Complex DSLs:

• ScalaTests supports various writing styles of tests and assertions. We had no idea how to choose the best style for our team. To simplify this, uTest has picked only one of the styles from ScalaTests, but it’s still a new domain-specific language on top of Scala. Specs2 introduces its own testing syntax, and even the very first example can be cryptic for new people, resulting in high learning cost.
• With these rich testing frameworks, using a consistent writing style is challenging as they have too much flexibility in writing test; Using rich assertion syntax like x should be (>= 0), x.name must_== "me", Array(1, 2, 3) ==> Array(1, 2, 3)needs practices and education within team members. We can force team members so as not to use them, but having a consensus on what can be used or not usually takes time.

Too minimalistic framework:

• On the other hand, a minimalist approach like minitest, which uses a limited set of syntax like asserts and test("...."), is too restricted. For example, I believe assertion syntax like x shouldBe y is a good invention in ScalaTest to make clear the meaning of assertions to represent (value) shoudlBe (expected value). In minitest assert(x == y) has the same meaning, but the intention of test writers is not clear because we can write it in two ways: assert(value == expected) or assert(expected == value). Minitest also has no feature selecting test cases to run by specifying test names; It only supports specifying class names to run, which is just a basic functionality of sbt.
• A minimalist approach forces us to be like a Zen mode. We can extend minitest with rich assertions, but we need to figure out the right balance between a minimalist and developing a DSL for our own teams.

AirSpec: Writing Tests As Functions In Scala

So where is a middle ground in-between these two extremes? We usually don’t want to learn too complex DSLs, and also we don’t want to be a minimalist, either.

Why can’t we use plain Scala functions to define tests? ScalaTest already has RefSpec to write tests in Scala functions. Unfortunately, however, its support is limited only to Scala JVM as Scala.js does not support runtime reflections to list function names. Scala.js is powerful for developing web applications in Scala, so we don’t want to drop the support for it. Probably the lack of runtime reflection in Scala.js is a reason why existing testing frameworks needed to develop their own DSLs like test(....) { test body } .

Now listing functions in Scala.js is totally possible by using airframe-surface, which is a library to inspect parameters and methods in a class by using reflection (in Scala JVM) or Scala macros (in Scala.js). So it was good timing for us to develop a new testing framework, which has more Scala-friendly syntax.

If we can define tests by using functions, it becomes possible to pass test dependencies through function arguments. Using local variables in a test class has been the best practice of setting up testing environments (e.g., database, servers, etc.), but it is not always ideal as we need to properly initialize and clean-up these variables for each test method by using setUp/tearDown (or before/after) methods. If we can simply pass these service instances to function arguments using Airframe DI, which has strong support of life-cycle management, we no longer need to write such setUp/tearDown steps for configuring testing environments. Once we define a production-quality service with proper lifecycle management hooks (using Airframe design and onStart/onShutdown hooks), we should be able to reuse this lifecycle management code even in test cases.

AirSpec was born with these ideas in mind by leveraging Airframe modules like Airframe Surface and DI. After implementing basic features of AirSpec, we’ve successfully migrated all of the test cases in 20+ Airframe modules into AirSpec, which were originally written in ScalaTest. Rewriting test cases was almost straightforward as AirSpec has handy shouldBe syntax and property testing support with ScalaCheck.

In the following sections, we will see how to use AirSpec to write tests in a Scala-friendly style.

Quick Start

To use AirSpec, add airspec to your test dependency and add wvlet.airspec.Framework as a TestFramework:

build.sbt

(Check the latest version on our web site):

libraryDependencies += "org.wvlet.airframe" %% "airspec" % "(version)" % "test"
testFrameworks += new TestFramework("wvlet.airspec.Framework")

For Scala.js, use %%%:

libraryDependencies += "org.wvlet.airframe" %%% "airspec" % "(version)" % "test"

Writing Unit Tests

In AirSpec test cases are defined as functions in a class (or an object) extending AirSpec. All public methods in the class will be executed as test cases:

import wvlet.airspec._
class MyTest extends AirSpec {
  // Basic assertion
  def emptySeqSizeShouldBe0: Unit = {
    assert(Seq.empty.size == 0)
  }
  // Catch an exception
  def emptySeqHeadShouldFail: Unit = {
    intercept[NoSuchElementException]{
      Seq.empty.head
    }
  }
}

If you need to define utility methods in a class, use a private or protected scope.

AirSpec supports basic assertions like assert, fail, ignore, cancel, pending, skip, intercept[E], and shouldBe matchers, which is will be explained later.

Tests in AirSpec are just regular functions in Scala. AirSpec is designed to use pure Scala syntax as much as possible so as not to introduce any complex DSLs, which are usually hard to remember.

AirSpec also supports powerful object lifecycle management, integrated with Airframe DI. The function arguments of test methods will be used for injecting objects that are necessary for running tests, and after finishing tests, these objects will be discarded properly.

Running Tests in sbt

AirSpec supports pattern matching for running only specific tests:

$ sbt
> test                                   # Run all tests
> testOnly -- (pattern)                  # Run all test matching the pattern (class name or test name)
> testOnly -- (class pattern)*(pattern)  # Search everything

pattern is used for partial matching with test names. It also supports wildcard (*) and regular expressions. Basically, AirSpec will find matches from the list of all (test class full name):(test function name) strings. Cases of test names will be ignored in the search.

Writing Specs In Natural Languages

If you prefer natural language descriptions for your test cases, use symbols for function names:

import wvlet.airspec._
class SeqSpec extends AirSpec {
  def `the size of empty Seq should be 0`: Unit = {
    assert(Seq.empty.size == 0)
  }
  // Catch an exception
  def `throw NoSuchElementException when taking the head of an empty Set`(): Unit = {
    intercept[NoSuchElementException] {
      Seq.empty.head
    }
  }
}

It is also possible to use symbols for test class names:

import wvlet.airspec._
class `Seq[X] test spec` extends AirSpec {
  def `the size of empty Seq should be 0`: Unit = {
    assert(Seq.empty.size == 0)
  }
}

shouldBe matchers

AirSpec supports handy assertions with shouldBe or shouldNotBe:

import wvlet.airspec._
class MyTest extends AirSpec {
  def test: Unit = {
    // checking the value equality with shouldBe, shouldNotBe:
    1 shouldBe 1
    1 shouldNotBe 2
    List().isEmpty shouldBe true
    // For optional values, shouldBe defined (or empty) can be used:
    Option("hello") shouldBe defined
    Option(null) shouldBe empty
    None shouldNotBe defined
    // For Arrays, shouldBe checks the equality with deep equals
    Array(1, 2) shouldBe Array(1, 2)
    // Collection checker 
    Seq(1) shouldBe defined
    Seq(1) shouldNotBe empty
    Seq(1, 2) shouldBe Seq(1, 2)
    (1, 'a') shouldBe (1, 'a')
    // Object equality checker
    val a = List(1, 2)
    val a1 = a
    val b = List(1, 2)
    a shouldBe a1
    a shouldBeTheSameInstanceAs a1
    a shouldBe b
    a shouldNotBeTheSameInstanceAs b
  }
}

Dependency Injection with Airframe DI

AirSpec can pass shared objects to your test cases by using function arguments. This is useful for sharing objects initialized only once at the beginning with test cases.

Global and Local Sessions

AirSpec manages two types of sessions: global and local:

• For each AirSpec instance, a single global session will be created.
• For each test method in the AirSpec instance, a local (child) session that inherits the global session will be created.

To configure the design of objects that will be created in each session, override configure(Design) and configureLocal(Design)methods in AirSpec.

Session LifeCycle

AirSpec manages global/local sessions in this order:

DI Example

This is an example to utilize a global session to share the same service instance between test methods:

import wvlet.airspec._
import wvlet.airframe._
case class ServiceConfig(port:Int)
class Service(config:ServerConfig) extends LogSupport {
  @PostConstruct
  def start {
    info(s"Starting a server at ${config.port}")
  }
  @PreDestroy
  def end {
    info(s"Stopping the server at ${config.port}")
  }
}
class ServiceSpec extends AirSpec with LogSupport {
  override protected def configure(design:Design): Design = {
    design
      .bind[Service].toSingleton
      .bind[ServiceConfig].toInstance(ServiceConfig(port=8080))
  }
  def test1(service:Service): Unit = {
     info(s"server id: ${service.hashCode}")
  }
  def test2(service:Service): Unit = {
     info(s"server id: ${service.hashCode}")
  }
}

This test shares the same Service instance between two test methods test1 and test2, and properly start and closes the service before and after running tests.

> testOnly -- ServiceSpec
2019-08-09 17:24:37.184-0700  info [Service] Starting a server at 8080  - (ServiceSpec.scala:25)
2019-08-09 17:24:37.188-0700  info [ServiceSpec] test1: server id: 588474577  - (ServiceSpec.scala:42)
2019-08-09 17:24:37.193-0700  info [ServiceSpec] test2: server id: 588474577  - (ServiceSpec.scala:46)
2019-08-09 17:24:37.194-0700  info [Service] Stopping the server at 8080  - (ServiceSpec.scala:30)
[info] ServiceSpec:
[info]  - test1 13.94ms
[info]  - test2 403.41us
[info] Passed: Total 2, Failed 0, Errors 0, Passed 2

Reusing Test Classes

To reuse test cases, create a fixture, which is a class, object, or trait extending AirSpec. Then call AirSpecContext.run(AirSpec instance), which can be injected to test method arguments:

import wvlet.airspec._
class MySpec extends AirSpec {
  // A template for reusable test cases
  class Fixture[A](data: Seq[A]) extends AirSpec {
    override protected def beforeAll: Unit = {
      info(s"Run tests for ${data}")
    }
    def emptyTest: Unit = {
      data shouldNotBe empty
    }
    def sizeTest: Unit = {
      data.length shouldBe data.size
    }
  }
  // Inject AirSpecContext using DI
  def test(context: AirSpecContext): Unit = {
    context.run(new Fixture(Seq(1, 2)))
    context.run(new Fixture(Seq("A", "B", "C")))
  }
}

This code will run the same Fixture two times using different data sets:

AirSpecContext also contains the name of test classes and method names, which would be useful to know which tests are currently running.

Property-Based Testing with ScalaCheck

Optionally AirSpec can integrate with ScalaCheck. Add wvlet.airspec.spi.PropertyCheck trait to your spec, and use forAll methods.

build.sbt

# Use %%% for Scala.js
libraryDependencies += "org.scalacheck" %% "scalacheck" % "1.14.0" % "test" 

import wvlet.airspec._
class PropertyBasedTest extends AirSpec with PropertyCheck {
  def testAllInt: Unit = {
    forAll{ i:Int => i.isValidInt shouldBe true }
  }
  def testCommutativity: Unit = {
    forAll{ (x:Int, y:Int) => x+y == y+x }
  }
  // Using custom genrators of ScalaCheck
  def useGenerator: Unit = {
    import org.scalacheck.Gen
    forAll(Gen.posNum[Long]){ x: Long => x > 0 shouldBe true }
  }
}

Scala.js

To use AirSpec in Scala.js, scalaJsSupport must be called inside your spec classes:

import wvlet.airspec._
class ScalaJSSpec extends AirSpec {
  // This is necessary to find test methods in Scala.js
  scalaJsSupport  
  def myTest: Unit = assert(1 == 1)
}

This is because Scala.js has no runtime reflection to find methods in AirSpec classes, so we need to provide method data by calling scalaJsSupport. Internally this will generate MethodSurfaces (airframe-surface) at compile-time, so that AirSpec can find test methods at runtime. Calling scalaJsSupport has no effect in Scala JVM platform, so you can share the same test code with Scala and Scala.js.

Conclusions

AirSpec is a new testing framework based on Scala-friendly code syntax without introducing any complex DSLs, which usually require additional learning cost. With the integration with Airframe DI and ScalaCheck, it also works as a full-fledged testing framework for advanced users.

The design philosophy of AirSpec is as follows:

• Minimizing the learning cost by using plain Scala functions as test cases and a limited set of assertion syntax like assert(cond) and x shouldBe y.
• Use function arguments to pass dependencies of your tests.
• Help users to select test cases to run using a handy keyword search.

The initial feature-complete version of AirSpec is 19.8.4. AirSpec is a part of 20+ Airframe modules and used for testing themselves. Currently, we have no intention to introduce furthermore assertion syntax to avoid becoming a complex framework. The next target is enhancing AirSpec to show more informative test results to users, like power assertions, test result differences, etc.

With AirSpec, you can now use plain Scala both for coding and writing tests. Let’s enjoy programming in Scala!

wvlet/airframe