Scalable Integration Testing Method

scalable testing
'
/ \\
/ \ \
/ \' \ <--------------------+ How do we
/ STn \'' \ | scale from
/ \ ' '\ | ST1 to STn?
/ \'' \ +-------------
/=============\ ' ' \ |
/ \ '' \ |
/ ST1 \' ' ' ' \ <--------+
/ \' ' ' ' \
/ \' ' ' ' \
/=======================\ ' ' \
/ \ ' ' ' /
/ Unit Tests \ ' / <--- Not in the scope
/ \' /
/===============================\/

The Scalable Method

Here’re key ideas forming the scalable method.

Test against a “complete” environment

Since the method doesn't distinguish ST1 and STn, the first prerequisite would be a complete environment.

Use the empty state as the baseline

Many of us are familiar with the test fixture, which is the pre-populated data we produced manually. With test fixture before each test case run, we can assume that certain business entities are already there, so when writing test cases we can reference an entity with a magic “id”, and start from there.

  • A fixture is fixed, and limited. There’re always complicated enough cases you have to introduce new fixtures. With more cases, the fixture set keeps growing, until it encounters the following “roofs”.
  • A fixture is a mental overhead that can not scale. When fixture set grows big enough, it becomes hard to memorize, and one has to frequently go back to DB/UI to lookup what entity is behind the magic number mentioned in a test case.
  • A fixture is a technical overhead that can not scale. When fixture set grows, the cleanup cost after each test case grows exponentially. In a typical system, in order to restore to a non-empty baseline, you can imagine that first DB tables need to be truncated, and then have fixture data re-injected. And derived storage subsystems, including search servers, Redis queues, and in-memory storages need to follow the exact same procedure. Also, it is important to point out that not all of those injected data will be used by every test case, which means significant time was spent on unnecessary IOs.
DB              Search
+-------+ +-------+ the cost of "wipe-and-seed"
| | derive | | | grows exponentially with
|-------| from |-------| | the size of state
| state | <----- | state | +--------------------
+-------+ +-------+

Build subsystems with test friendliness in mind

Subsystem
+-----------+
| :inspect_port <--------- Call to Inspect
| |
| :reset_port <--------- Call to Reset
+-----------+

Reuse common test components

There’re a few components that can be made reusable so developers can write test easier and faster.

  • Factory (for data preparation)
  • Programmable Mock (for limiting the test boundary to a smaller scale)
  • Subsystem Specific Utils
# language-less, just assume the language would support method call, and somehow allow passing an option map as a parametercreate("user", { "active": true })# instead ofINSERT INTO `user` (`id`, `name`, `status`) VALUES (1, ‘Marry’, ‘ACTIVE’);
# the order factory should created depended payment and address automatically
create("order")
# instead ofINSERT INTO `payment` (`id`, `type`, `amount`) VALUES (1, ‘VISA’, 100);
INSERT INTO `address` (`id`, `line1`, `line2`) VALUES (1, '1 Main St', 'Apt 5');
INSERT INTO `order` (`id`, `payment_id`, `address_id`) VALUES (111, 1, 1);
  • by making API calls (performant and stable)
  • by simulating user clicks on UI (slow but stable)
  • by injecting into storage engine such as DB directly (very performant but also very brittle, since it is highly bound to implementation detail)
# When mock is off, mock acts as a reverse proxy+----------+   1   +----------+   2   +----------+ 
| | ----> | | ----> | |
| Consumer | | Mock | | Provider |
| | <---- | | <---- | |
+----------+ 4 +----------+ 3 +----------+
# When mock is on+----------+ 1 +----------+ +----------+
| | ----> | | | |
| Consumer | | Mock | | Provider |
| | <---- | | | |
+----------+ 2 +----------+ +----------+
# callers should always point to mock server, which is preconfigured to proxy to the actual serviceGET /users
HOST mock-user-service
# default relay to https://provider.com/usersGET /users
HOST provider.com
# turn on mockPOST /_mock?url=/users
HOST: mock-service
Body: {“response_code”: “200”, “response_headers”: [], “response_body”: “…user list…” }
# turn off mockPOST /_unmock?url=/users
HOST: mock-service
  • a function that simulates login can be reused by all UI integration cases
  • a function checking background queue status can be reused by even cross subsystem test cases

Drawbacks

The scalable method comes with its own complexities and constant overhead.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store