Bumble Tech
Published in

Bumble Tech

article image: pyramid with layers
Photo by Jeremy Bishop on Unsplash

Hourglass into Pyramid: how you can improve the structure of your tests


What is a Test Pyramid?

  • Tests are required for each of its different levels;
  • The higher up the pyramid you go, the fewer tests you should have.
  • Write lots of small and fast unit tests
  • Write some middle-level tests
  • Write very few high-level end-to-end tests

iOS Test Levels at Bumble explained

  • Manual Tests — checking the functionality without the help of autotests. Currently, these activities are limited to a handful of cases and we try to automate the most repetitive and tedious parts of manual testing as much as we can. However, we still have them. For instance, when a new feature is introduced (covered by autotests, of course): Developers and QAs have a so-called “1-hour QA Session”. The goal here is to ensure the feature is made according to the agreed-upon standards and ready to be shipped. Moreover, since our apps are released every week, prior to shipping them we perform manual release testing. We check the most important functionality that can’t be automated or that requires additional attention. Finally, we have various activities, like Testing Dojos that involve collective testing of the apps by numerous people from various teams.
  • End-To-End Tests — these are black-box tests written by our QA engineers that focus on the client’s integration with other Bumble services and infrastructure. We use Calabash, a cross-platform automation framework. Tests are written on Ruby and use Cucumber and Gherkin syntax. In my previous article, I lifted the curtain on some of the process patterns we follow during the development of end-to-end tests. What’s more, we have published a sample project to showcase our framework, which you can find here. We also have a subset of tests that go through the app and take screenshots of every screen, across various device configurations and languages. We call these tests — Liveshots, and you can learn more about them here.
  • Component tests and Component Integration tests — these are particular types of black-box testing with test cases based on the specifications of the feature components, and their integration both between each other and with iOS services. They are meant to be isolated from the rest of Bumble systems. They are written in Swift (like the other tests in the lower levels of the pyramid) and rely on an in-built XCTest framework. We will return to them later. For this article, I’ll combine Component tests and Component Integration tests into one category named Component tests. This is partially for simplicity because currently, we have them both defined on the application level. In future, we intend Component tests to be moved to specific feature modules.
  • Visual regression Tests (VRT) — these are tests that check that UI design matches expectations and integrates UI components correctly with the OS UI subsystem. Often these types of tests are called “Snapshot tests”.
  • Unit Tests — these tests ensure that a unit meets its design and behaves as intended. A unit is often an entire interface, such as a class, but it could also be an individual method. They should ensure that all non-trivial code paths are tested (including happy path and edge cases).
  • Lack of isolation
  • Asynchronous behaviour
  • Remote services
  • Resource leaks

Building the Test Pyramid

  1. Test coverage and levels should be discussed at the planning stage. As soon as the new feature is about to be started, it gets an assigned QA. They actively participate in the feature’s kick-off together with developers and all other parties involved. Post kick-off and having reviewed all the documentation in detail, the QA and developer start working on the various test scenarios. This is the time when they discuss and agree on the proper test coverage for the feature. Only when this has happened coding for both features and tests can start.
  2. Tests may be of different granularities, but they should always add value. Remember: the higher the level of the tests, the fewer tests there should be. Therefore, we are pushing the tests as far down the test pyramid as we can.
    Note: when a higher-level test gives more confidence in the application working correctly, it should be added. Otherwise, it is better to stick to the lower levels.
  3. Treat code of the autotests similarly to the application under test code. Code of the autotests should get the same level of attention. Otherwise, we risk ending up in a situation where we have an unmaintainable tests codebase. That won’t be much help to us and will require a great deal of effort to work with.

Writing tests on a lower level

Moving end-to-end tests to Component level

  • During development, the tests are run against the real server and interactions with the server are recorded and stored.
  • After the feature is merged, all the tests are run in playback mode only.
  • Later, if the test needs amending — it needs to be re-recorded. Before selecting this approach we carefully considered all its pros and cons.
  • Components external to the iOS system, such as network, server and Apple Push Notification service (APNs). They are checked in end-to-end tests.
  • User interface design, layout and appearance. These are checked in Visual regression Tests.
  • Logic is defined in a set of classes without a user interface or a single class. (Unit Tests)