Standardising DataSource implementations through testing

Background

ExoPlayer has a modular structure, which allows applications using the library to override or vary many different aspects of its behaviour. This modularity is achieved through the use of interfaces that represent distinct parts of the media loading and playback pipeline. Examples include DataSource, MediaSource, TrackSelector, Extractor and Renderer. For more information about how these components fit together in the library, please see our glossary.

In some cases where ExoPlayer defines an interface for a piece of player functionality, it only provides a single implementation (e.g. DefaultTrackSelector), and the interface only exists to allow applications to write a fully-custom implementation. In other cases (e.g. DataSource, MediaSource, etc.) ExoPlayer provides many different implementations, and applications are of course able to provide fully custom implementations too. In particular we have a large number of DataSource implementations in the library, and we also know this is a popular component for applications to customise because we often receive questions about it on our issue tracker.

The problem

The DataSource interface, despite having relatively few methods, is easy to implement incorrectly. Before the work described below it was hard to check a DataSource implementation for correctness. Depending on the problem, an incorrect DataSource implementation can seem to work fine during simple playback and then introduce surprising behaviour in specific circumstances (e.g. seeking).

We noticed that among our implementations of DataSource there were some subtle (but important) differences in behaviour, often around edge cases such as reading the last byte in a file or stream. We also found that the implementations that app developers shared with us in GitHub issues were sometimes incorrect (e.g. ignoring DataSpec#position).

Being able to test a DataSource implementation for correctness, including exercising edge cases that are hard to reproduce in real playbacks, would allow us to both improve our own implementations and also help app developers to improve theirs.

The solution

We decided to build a ‘contract test suite’ to check behaviours that should be common to all DataSource implementations. We designed this to allow the same assertions to be easily applied to many different implementations.

DataSourceContractTest is an abstract class containing several JUnit4 @Test methods, and some overridable methods that allow subclasses to configure the backing data and expected results.

We now have 13 subclasses of this test in the library, with most running under Robolectric and the rest running as instrumented tests (because Robolectric doesn’t provide enough fidelity for the system being tested). FileDataSourceContractTest shows a relatively simple example, while RawResourceDataSourceContractTest is more complicated because it tests different URI structures.

Impact

In the process of adding these tests we found and fixed some bugs in our implementations (example). We also tightened some previously undefined behaviour when a DataSource is opened in a way that extends beyond the available data (documentation, test enforcement and implementation fixes).

Testing your own DataSource

We would encourage all library users who maintain their own DataSource implementations to add contract tests for them by subclassing DataSourceContractTest. If we add additional tests to the DataSourceContractTest, your subclasses will pick these up automatically when you upgrade your exoplayer-testutils dependency.

Below are some suggestions based on our experience of adding these tests to the ExoPlayer library. You can also browse the examples in the library directly for further inspiration.

HTTP DataSource implementations

If your DataSource reads from an HTTP connection, you might find our HttpDataSourceTestEnv test rule helpful. It provides a set of standard test resources and configures Square’s MockWebServer to ‘serve’ these realistically, including support for redirection and range requests. DefaultHttpDataSourceContractTest shows an example of using this rule to write a DataSourceContractTest implementation.

Delegating DataSource implementations

If you’re testing a ‘wrapper’ or ‘delegating’ DataSource, consider using FakeDataSource as the inner implementation in your test, it’s designed to simulate various different DataSource behaviours. You can look at CacheDataSourceContractTest for an example of this.

Suppressing failures

The assertions in the test suite should generally pass for all implementations of DataSource, and a failure usually indicates a bug in the DataSource implementation. However, there are some cases where suppressing a specific test may be necessary. For example:

• Some tests exercise behaviour that can’t always be recreated by the underlying technology (e.g. non-existent resources in ByteArrayDataSourceContractTest).
• Some implementations may have known issues that should be resolved in future, but for now need to be ignored (e.g. UdpDataSourceContractTest suppresses tests related to DataSpec#position and DataSpec#length).

As you can see from these examples, a failing test can be suppressed by overriding the @Test method with an empty implementation. We recommend you also annotate this @Ignore so JUnit will correctly record it as ignored (instead of always passing), but this isn’t strictly necessary.

Don’t mix with implementation-specific tests

JUnit4 discourages the use of inheritance (in contrast to JUnit3, which requires it). We’ve ignored this recommendation when designing this test suite because inheritance offers a convenient way to share the test assertions themselves (rather than setup/teardown, which is better shared through rules).

We understand that inheritance in test classes can be confusing. In particular it’s not always obvious where a test is defined. In order to reduce the risk of this confusion, we’d encourage you to keep implementation-specific assertions in a separate test class and not to put additional @Test methods in your DataSourceContractTest subclass (e.g. ExoPlayer has both CacheDataSourceTest and CacheDataSourceContractTest). This means that there’s only one class in each hierarchy responsible for defining the @Test methods.

Please feel free to get in touch via our issue tracker if you have any questions or encounter problems with subclassing DataSourceContractTest. Thanks for reading!