QA Automation: How to start and define your roadmap

Clovis Bernier
Jul 28 · 7 min read

In a first article, we detailed how we were able to Improve your app quality with simple QA processes, taking the example of BlaBlaCar Daily, the commute carpooling app by BlaBlaCar.

6 months later, things have evolved, and we decided to start implementing QA Automation processes. In this article, we’ll quickly go over the reason why we decided to start automating, and we’ll focus on how we prioritised which test cases to automate first.

Why we decided to start automating our tests

After 4 years of development, BlaBlaCar Daily has gathered more and more features, thus more and more user flows. This means a growing number of test cases for us to take care of manually as QA Specialists. Today, we are able to maintain a high level of confidence at every release, but at this pace, our Submission Test Session will get heavier and heavier, as the number of Regression Test Cases keeps increasing. That is the main reason why we decided to start automating some of those Regression Test Cases (i.e. the most critical flows of our app, the ones we don’t want to see failing in production, as explained in our first article).

How we started automating

BlaBlaCar Daily is only available on apps (Android and iOS). Given that the release process on these two platforms is long, the cost of a rollback is high. To be more precise, it takes a few days for a new app version to be available for our members from when we decide to release it. This means that if a major bug is discovered in production, it takes time to release the fix, or to rollback to the previous version which doesn’t contain the bug. This is the main reason why we decided to focus our automation work on the front end side of our service, and more precisely on the creation of automated UI Tests, for Android and iOS.

We chose the native solution, which means that we don’t need a third-party framework that would create an overlay. In easier terms, we will directly code our UI tests in the platforms’ respective languages (XCUITests for iOS, and Espresso for Android). We also decided to mock the backend behaviours. If you’re not familiar with the concept of mocking, it simply consists in mimicking an object, in order to test another one.

Here are the advantages of these decisions:

For the disadvantages:

Now that we’ve explained why and how we started automating our tests at BlaBlaCar Daily, let’s deep dive on our homemade technique to create our QA Automation roadmap and, more precisely, which test cases to automate first.

How we decided which test cases to automate first

At the moment of writing this article, BlaBlaCar Daily has 265 test cases, among which 51 are Regression Test Cases. Like for BlaBlaCar, the Regression Test Cases are the ones we want to see automated. We had to come up with a way to prioritise them, and this is what we most wanted to share in this article.

So, we sat down with the objective to define which criteria we should make our decision on. We came up with the following two criteria, which we confronted in order to create a ratio:

For the first one, we used the T-shirt sizing estimation method, applying a ponderation based on the Fibonacci sequence, which we started at 2.

If you feel lost after reading the previous sentence, no worries! Here are simple definitions of the two concepts:

Here is what our chart for the ‘Difficulty to Test Manually’ looks like:

XS → 2

S → 3

M → 5

L → 8

XL → 13

For the ‘Difficulty to Automate’, we applied the following ponderation:

Hard → 30

Medium → 20

Easy → 10

With the QA Team, we went over the 51 most critical test cases, and sized each of them, discussing the difficulty to test manually and the difficulty to automate.

For the ‘Difficulty to Test Manually’, we sized each test case depending on the following: time to test, setup and testability requirements and feature documentation.

For the ‘Difficulty to Automate’, we sized each test case depending on the following: time and difficulty to automate (for example taking into account if there is a third-party dependency).

Then, we created the ratio, by dividing the ‘Difficulty to Automate’ by the ‘Difficulty to Test Manually’. And we were done. Now we had our roadmap!

Indeed, the test cases with the higher ratio are the ones we will automate first.

Example with some of our test cases

In order to better illustrate our roadmap ranking technique, we picked 4 of our test cases:

1- As a member, I can add a credit card

2- As a registered member, I can modify my personal info

3- As a driver, I can accept a passenger’s ride request

4- As a future BlaBlaCar Daily member, I can send a request from Google Maps

Let’s weigh each of them with our 2 criteria.

1- As a member, I can add a credit card

With this one, we want to test that our members can add a credit card to their account. The flow is both easy to test manually and to automate, that’s why we respectively weighed it as XS (2) and Easy (10).

2- As a member, I can modify my personal info

For this one, we want to test that a member can access their profile and modify their personal info (name, email address, date of birth and company). This test is not difficult to test manually, but it takes some time, given it contains many steps. So, we sized the ‘Difficulty to Test Manually’ as S (3). On the other hand, it’s not difficult to automate, as it simply consists in modifying information on the personal information page of the profile. So, we sized the ‘Difficulty to Automate’ as Easy (10).

3- As a driver, I can accept a passenger’s ride request

For the third one, the ‘Difficulty to Test Manually’ is M (5), because we have to create two accounts: one as a passenger who sends the request, and one as a driver who accepts it. It’s not complicated, but it’s time consuming. Concerning the ‘Difficulty to Automate’, we set it at Medium (20), given we’ll need to create the ride request that the driver will receive. For this, we will mock the ride request.

4- As a future BlaBlaCar Daily member, I can send a request from Google Maps

For this last test case, the ‘Difficulty to Test Manually’ and ‘Difficulty to Automate’ are both at the highest level, at XL (13) and Hard (30), respectively. Indeed, testing the flow of creating a ride request from Google Maps (where BlaBlaCar Daily results come up in the transport tab) to our commuting app is a huge project that is both difficult to test manually and to automate. That is the reason why we allocated the higher weight for both criteria.

Now, let’s calculate the ratios, by dividing the ‘Difficulty to Automate’ by the ‘Difficulty to Test Manually’:

1– 10 / 2 = 5

2– 10 / 3 = 3.3

3– 20/4 = 5

4– 30/13 = 2.3

And finally, let’s rank them, the higher ratio being the one to start automating first:

3- As a driver, I can accept a passenger’s ride request (5)

1- As a member, I can add a credit card (5)

2- As a member, I can modify my personal info (3.3)

4- As a future BlaBlaCar Daily member, I can send a request from Google Maps (2.3)

You’ve noticed that test cases 1 and 3 have the same ratio (5), and you may wonder why we would start with 3 and not 1. As mentioned above, the 1st test case is easy to test manually, so we would be less bothered if it is not automated, compared to the 3rd test case. And that’s it! By applying this technique to all of our Regression Test Cases, we were able to know where to start and what to do next!

This is the first step of our automation work at BlaBlaCar Daily and we hope to increase our efficiency in the months to come by automating as many regression test cases as possible. We hope you found this article useful, and please do not hesitate to reach out to us at qa@blablacar.com if you have questions or ideas for improvement!

BlaBlaCar

The stories behind BlaBlaCar, the world’s leading multimodal mobility platform.

BlaBlaCar

BlaBlaCar is the go-to marketplace for shared mobility, combining short and long-distance carpooling and buses. In building the future of mobility, we set ourselves high and ambitious targets and bring tech and data to the heart of our product experience and strategy.

Clovis Bernier

Written by

Quality Assurance Specialist @BlaBlaCar

BlaBlaCar

BlaBlaCar is the go-to marketplace for shared mobility, combining short and long-distance carpooling and buses. In building the future of mobility, we set ourselves high and ambitious targets and bring tech and data to the heart of our product experience and strategy.