Better Parameterized Tests with Burst

An alternate data variation mechanism for JUnit tests.

Square Engineering
Oct 21, 2014 · 4 min read

Written by Daniel Lubarov, Jake Wharton, and D. Koutsogiorgas.

Heads up, we’ve moved! If you’d like to continue keeping up with the latest technical content from Square please visit us at our new home https://developer.squareup.com/blog

At Square, we invest a lot of effort in testing to ensure that our software is reliable. Not only do we unit test our business logic, we also use automated UI tests to detect bugs and prevent regressions in our applications. We also use manual testing and phased rollouts as additional safeguards.

Sometimes we want to repeat a test several times with different parameters. If we’re testing a web server for example, we might want to repeat a request using different versions of the HTTP protocol.

Test authors often use foreach loops to enumerate test parameters. When such a test fails, the investigation usually starts with determining the parameter or parameters which triggered the failure. In the interest of making tests easier to maintain, it is helpful to have a framework which can immediately show you the parameters with which a test failed.

JUnit’s Parameterized

@RunWith(Parameterized.class)
public class WebServerTest {
enum Protocol { HTTP_1_0, HTTP_1_1, HTTP_2 }
@Parameters(name = "protocol={0}")
public static Collection<Object[]> data() {
return Arrays.asList(
new Object[] { Protocol.HTTP_1_0 },
new Object[] { Protocol.HTTP_1_1 },
new Object[] { Protocol.HTTP_2 }
);
}
private final Protocol protocol; public WebServerTest(Protocol protocol) {
this.protocol = protocol;
}
@Test public void testGetRequest() {
dispatchGetRequest(protocol);
}
@Test public void testPostRequest() {
dispatchPostRequest(protocol);
}
}

Parameterized will generate the following hierarchy of tests, making it easy to see where the failure occurred:

Image for post
Image for post

Parameterized is useful, but declaring each combination of test parameters can be cumbersome. They are declared using object arrays, which are awkward to use. Additionally, there is no way to parameterize a single test method (apart from moving it into a separate class).

Finally, since Parameterized is part of the JUnit 4 framework, it is not available in JUnit 3. While most modern projects have adopted JUnit 4, Android’s test framework still requires JUnit 3.

Introducing Burst

@RunWith(BurstJUnit4.class)
public class WebServerTest {
enum Protocol { HTTP_1_0, HTTP_1_1, HTTP_2 }
private final Protocol protocol; public WebServerTest(Protocol protocol) {
this.protocol = protocol;
}
@Test public void testGetRequest() {
dispatchGetRequest(protocol);
}
@Test public void testPostRequest() {
dispatchPostRequest(protocol);
}
}

Running this will produce a hierarchy of tests similar to the one produced by Parameterized:

Image for post
Image for post

In addition to class-level parameters, Burst also supports parameters at the method level. This can be used standalone or in tandem with class parameters. Here’s an example with both class and method parameters:

@RunWith(BurstJUnit4.class)
public class WebServerTest {
enum Protocol { HTTP_1_0, HTTP_1_1, HTTP_2 }
enum Method { GET, PUT, POST }
private final Protocol protocol; public WebServerTest(Protocol protocol) {
this.protocol = protocol;
}
@Test public void testConnect() {
connect(protocol);
}
@Test public void testRequest(Method method) {
dispatchRequest(protocol, method);
}
}

And the resulting test hierarchy:

Image for post
Image for post

If you declare more than one enum parameter in a constructor or method, Burst will generate a test for each unique combination of parameters.

Sometimes your test parameter isn’t a single value that can trivially be represented by an enum. But keep in mind that enums are flexible — you can embed whatever data you need in them. If you’re testing some logic that involves credit cards and want to repeat a test using several cards, you could declare:

enum CreditCard {
VISA("4111111111111111", "1804", "123"),
MASTERCARD("5500005555555559", "1804", "456");
final String accountNumber;
final String expirationYYMM;
final String securityCode;
CreditCard(String accountNumber, String expirationYYMM, String securityCode) {
this.accountNumber = accountNumber;
this.expirationYYMM = expirationYYMM;
this.securityCode = securityCode;
}
}

Android

Like our JUnit 4 runner, BurstAndroid lets you add enum parameters to test constructors. Method-level parameters are unfortunately not supported, as Android’s test framework only includes zero-parameter methods when constructing a test suite.

Filtering tests

Our apps have certain features which only exist on phones or tablets, so we annotate associated tests with @PhoneOnly or @TabletOnly. BurstAndroid lets us skip these tests based on the device being tested.

Downloading

For more details on how we approach UI testing at Square, check out Dimitris’ Droidcon slides.

Square Corner Blog

Buying and selling sound like simple things - and they…

Square Engineering

Written by

The official account for @Square Engineering.

Square Corner Blog

Buying and selling sound like simple things - and they should be. Somewhere along the way, they got complicated. At Square, we're working hard to make commerce easy for everyone.

Square Engineering

Written by

The official account for @Square Engineering.

Square Corner Blog

Buying and selling sound like simple things - and they should be. Somewhere along the way, they got complicated. At Square, we're working hard to make commerce easy for everyone.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store