API for QA: Testing features when you have no access to code

There are lots of app features that can’t be tested quickly without changing the source code. Think of this typical task every developer has had to perform: three days after a user has registered, you want to offer them the option of purchasing premium access to the product at a discounted price. To test whether this promo is going to work, you need to register a test user and then wait three days until the promised dialog window appears. But clearly, a developer isn’t going to want to wait that long. They will simply amend the value for the constant in the code that sets the time interval, reducing it from 3 days to 20 seconds — and so get the result almost immediately.

Let’s say that the data for the promo offer is given by the server, while the promo is displayed on the client. If the server tester is working in an environment that allows them to fiddle with the code, they will be able to execute this task quickly. However, if the environment no longer offers the option of changing the code (staging, production), then there is a problem.

It goes without saying that, when reproducing a case, you can’t change the value right at staging or production. You could seek a solution by moving the value to a database, but this would give rise to the following three difficulties:

• the code would become significantly more complicated
• you would have to give all interested parties the rights to update the databases (and monitoring this), increasing the likelihood of human error, when working with the database, and risking a wide variety of negative consequences
• the value in the database would be the same for all users: it could no longer be changed for one particular colleague while remaining the same for everyone else.

Once the server task arrived at production it wouldn’t be the end of the story either. Neither the client developers nor tests simulating user behaviour, such as Calabash or Selenium, would have the option of changing server code.

We cannot allow ourselves the luxury of spending time, energy and resources on solving such problems when reproducing new functionality. So, we started to look for a way to influence the parameters of the system without changing the source code. There were five key criteria it would need to satisfy. It would have to be:

• simple for the user: all the engineers would be using it, regardless of their particular area of expertise
• suitable various types of test: Calabash, Selenium and functional tests
• Simple for the developer: it should not become more expensive to implement a feature as a result of being able to affect its parameters
• secure: however this works, it should not adversely affect real users at production
• Easy to document.

At Bumble this tool is called QAAPI. Based on its name, it will be clear that this was initially created to help testers, but it proved to be so powerful and multifaceted that nowadays there isn’t a single colleague involved in processes for creating new features and supporting existing ones who could envisage working without it.

In this article, I explain the QAAPI concept, and how it is implemented at Bumble Inc — the parent company operating Badoo and Bumble apps.

Using QAAPI

QAAPI is a set of methods called via HTTP and returning a JSON response. This format is suitable for everyone: a person can call methods from a browser while, at the same time, every kind of test (Calabash, Selenium etc.) can find a way of making an HTTP query. Similarly in the case of a JSON response: this can be read equally easily by a person or by tests.

For a QAAPI user, this is what the solution to the problem of showing a promotion after registration would look like. There is a method called SetPromoTimeOffset and a corresponding URL accepting, as a GET parameter, the number of seconds after registration. Next, the promotion needs to be shown along with the ID of the user for whom the new value will be used. For example:

/SetPromoTimeOffset?seconds=20&userid=12345

The query returns a short answer:

{ “success” : true }

And, 20 seconds after registration, the user with the ID 12345 will be shown the promotion.

How can this method be implemented?

Creating a QAAPI method

Every QAAPI method includes three elements:

• Description
• Input parameters
• Body of the method with logic.

All QAAPI parameters are strictly typified and are described by separate classes with their own validation. They can be simple, such as Boolean or DateTime, and specific, such as user_id or session_id for example.

A class for the SetPromoTimeOffset method would look like this:

class SetPromoTimeOffset extends \QAAPI\Methods\AbstractMethod
{
    public function getDescription() : string
    {
        return <<<Description
Sets a time offset in seconds between the user's registration date and the promo showing
Description;
    }
    public function getParamsConfig() : array
    {
        return [
            'user_id' => \QAAPI\Params\UserId::create(),
            'seconds' => \QAAPI\Params\PositiveInteger::create()->setDescription('Offset in seconds'),
        ];
    }
    public function run() : \QAAPI\Models\Output
    {
        // logic here
        return \QAAPI\Models\Output::success();
    }
}

The principle of how a method works depends on its purpose. The overwhelming majority of cases can be assigned to one of three categories:

• receiving data
• changing data
• changing server logic.

The first two categories assume simple interaction with the database. In order to change server logic, the relevant user needs to be flagged in some way their relevant values saved.

QAAPI security

QAAPI security is implemented on three levels. At the first level, developers have total control over what the method does, and its implementation is hidden from the end-user. There is no need to give anyone rights to the database or to monitor it. The QAAPI system itself requires authorisation, which in our case is implemented using Google authorisation for engineers and a secret token in the header for tests.

The second level of security is network access restriction to QAAPI: it needs to be used either from the office or using VPN.

And the third level is a decision in principle to allow QAAPI to work solely with users flagged as test users. This removes the risk, in the case of real users, of something being broken, and also of violating regulations around the handling of personal data such as GDPR, for example.

Documentation

The number of methods in our system has already passed the 1,500 marks. It goes without saying that it is impossible to keep them all in your head, so it was necessary to take the time to create detailed documentation with a search option and detailed descriptions. Having a strict description of each method allows this documentation to be generated ‘on the fly’.

This is what our new method looks like:

Example of the automatically generated documentation

In this way, we can also generate clients/classes for all possible tests (Calabash, Selenium etc.).

QAAPI at Bumble

Our experience shows that the only limitations in terms of the areas where QAAPI can be applied are the bounds of the developer’s creativity. QAAPI can be used in various settings:

• For registering a test user, when we have to specify gender, age, interests, as well as the number and type of photos
• For exchanging messages between users: we create a second user with the necessary parameters and send a message on their behalf (in their name)
• When activating services, reproducing the flow for verification of the user, verification of payment and so on.

You can combine methods, calling one from the body of another. Nor does this make a call more expensive; internal interaction between methods takes place locally and not via HTTP.

It would also have been a major oversight if the members of our team not using PHP were deprived of all the expressiveness of QAAPI and forced to content themselves with just ready-made methods. So, we created an auxiliary tool for them, that allows them to write scenarios in Lua and within them to call any combinations of QAAPI methods.

Unlike methods, scenarios are not to be found in a repository, so developers also use them to avoid creating unnecessary extra methods, and have the option of changing them ‘on the fly’. A scenario can also be its own kind of documentation for reproducing a particular feature and can be spread throughout the entire project flow from the server developer to the client tester/automation engineer.

Conclusion

The QAAPI concept is very simple to implement. It can be used in any project where you need to reproduce complicated cases and cover them with tests.

QAAPI gives us loads of benefits:

• it prepares data for automated and manual testing
• it ensures data on the state of the user/feature can be obtained quickly and securely
• it becomes an access point for studying how the functionality works (for example, where a method includes a service, it is easy to move from the body of the method to the code of the service itself)
• Scenarios de facto constitute documentation of features with a complex flow.

It usually takes no more than a couple of minutes to create a simple method, while QAAPI helps save a colossal amount of time. Besides the obvious benefits, a correctly written method or scenario means there is no need to focus on the details of a given feature, but on solving a task.

QAAPI has been used at Bumble since 2013, and today it is virtually impossible to imagine life without it. At the same time supporting this tool costs us practically nothing at all: documentation and classes/clients for tests are generated automatically, each method is autonomous and is independent of others.

We often refer to QAAPI, but to date, we have never talked in-depth about it in our articles. We hope that what we have shared here will inspire you to create a similar tool or to share your own unusual solutions with the community.