Problems with using default scenarios in acceptance tests
At Kayako we’re using ember-cli-mirage to run in development without needing the live API and to also run tests against the mock API.
Mirage has a default scenario, which provides the data used when running ember server in development. As Kayako is a fairly complex application, this default scenario has grown to over 1500 lines to setup the data needed to interact with every page of the app.
When writing an acceptance test, it’s quite tempting to just load up this default scenario and use that in the test, as there’s a fair few moving parts which need to be wired up in order to get to any given page before it can be tested.
This leads to a number of problems.
It makes the whole test suite slower
At first, when the default scenario was fairly simple, it didn’t add much overhead. As features have been added over time, every feature making a small addition to the default scenario, just loading the scenario adds nearly half a second to each test. Multiply that across every acceptance test and that’s a lot of time spent loading data which isn’t needed.
Displaying unnecessary components can be slow
We had a single test which was taking nearly 40s to run, testing a fairly complex order of operations which required visiting a set of pages in a particular order:
loadDefaultScenario();login();visit(‘/foo’);
andThen(function() { ... });visit(‘/foo/1’);
andThen(function() { ... });visit(‘/foo/1/bar’);
andThen(function() { ... });visit(‘/foo/1’);
andThen(function() { ... });visit(‘/foo/2/bar’);
andThen(function() { ... });
...
Each one of these pages being visited has an activity feed and a number of other components which have nothing to do with feature being tested, but because we’re using the default scenario they were all being fully populated with a large amount of data, slowing down the test.
Once we’d figured out the bare minimum required to display the screens, we were down to ~10 lines of setup, a far cry from the ~1500 lines previously being run, and the test ran 5 times faster.
It makes your tests brittle and hard to reason about
As all the tests are using the same scenario, it’s very difficult to know what data is required for each one and making small changes can break tests in far distant parts of the application.
If you’re reaching for the default scenario because it’s easier than setting up just what’s needed for the test at hand, that tells you something.
Far better would be to have a scenario which sets up the absolute bare minimum for the application to boot and then each test can describe just what it needs.
At Kayako we’re exploring a number of approaches to tidy up our test suite, making it easier to write new tests, easier to understand existing tests and improving performance as we go. We’ll be sure to report back with our findings!
