Crowd Testing: opportunity or scam for testers?

Have you ever heard of Crowd Testing (or Crowdsourced Testing)? Crowd Testing (not to be confused with Cloud Testing) is trending in the world of software testing, and uses the benefits and efficiency of crowdsourcing and cloud platforms.

This term is quite vague and can mean either asking feedback from users, or simply asking for advice on a new service or new version of the website. But this is also used to describe tests done by experts like you, dear readers. There are several cases, but the testers will generally be paid depending on the bugs found, or for the execution of specific test plans. The applications to be tested are all kinds of mobile applications, websites, but also e-commerce websites.

The tests are finally performed by several testers from different countries, using heterogeneous devices and with a large panel of configurations, languages, screen sizes, etc. In this article, we will take a quick tour of the various Crowd Testing services of the market, from the point of view of the tester offering his expertise (and not from the point of view of the company requesting his services). This is the story of my experience over a few months, and it reflects my own views only.

General

Many companies have entered the Crowd Testing market. I have tried some of them: TestIO, TesterWork, Crowd App Quality, We Are Testers, QAOnAir, UberTesters, BugFinders and uTest… but there are so many others, and it’s hard to be exhaustive because new comers arise frequently.

What can we test, for example?

• E-commerce sites (mobile or desktop)
• Various mobile applications (iOS or Android)
• A new feature, such as a loyalty program with points to earn
• A new and more intelligent address form (street suggestion, postal code, delivery point, etc.)
• Online payment services
• Chat bots
• Translations, localization
• …

In general, how is it going?

First of all, you have to be registered on the platform, which often requires a various number of criteria to meet or even a test to pass (QCM or an application to test with issues to file). Then, according to your skills and location, you will be invited to test cycles. You will have to accept (or not) to really be able to participate in the cycle. For these last stages, you will have to be extremely fast: it is even said that staying glued to your mailbox day and night is the best method to always be selected for the best cycles.

You will then be given a description of the mission, which may be to run test cases (with screenshots or videos to provide to prove that you have done well) or simply search for bugs in the requested perimeter. All of these instructions are sometimes unclear, often contain inconsistencies, and it is not always easy to get an answer within a reasonable time through the client. It will take a lot of patience.

When you have found a bug, you will first have to check that it is not a known bug. You’ll sometimes have a spreadsheet-style document listing the known bugs, and if you’re lucky these issues are even already populated in the bug management tool you’re asked to use, but it’s pretty rare. In addition, you must deal with the titles and descriptions provided, written by previous Crowd Testers in an approximate language and with a vocabulary not necessarily shared by all (yes, I know, I’m not always clear when writing in English too, I’m better with French). You will therefore inevitably enter bugs which you will not find any known reference for and which will be considered as duplicates at the end.

Then, if your bug is not in the history of the known bugs, it is necessary to be the first to record it. I happened to have a bug refused because another tester had recorded it 40 seconds before mine. In another case, we were a group of testers testing the application in French, another group in German, another group in Italian, etc. And my issue happened to be refused because an Italian tester had found and filed it a few hours before me. Oh! I forgot to mention that each group did not have access to the list of bugs reported by other groups of other languages. So you have to be very calm when your work is refused this way.

Your work is more or less monitored, and your reputation increases with your performance. So you’d better make good bug reports, with valuable information for the client because otherwise you will have a high rate of refusal which will not help you to be invited thereafter. Payments are then based on the actions performed (number of test cases played) or the number and value of bugs reported and accepted. Rates are variable depending on the case and announced from the start in a more or less clear way (many examples of bad surprises after the fact).

My experience

When I wanted to try the different platforms mentioned in this article, while I was unemployed and preparing for my future freelance activity, I wanted to devote a fortnight. My goal was to better understand what this type of platforms could bring for companies, but also what I could get, in terms of knowledge and salary. At the end of this short period, I realized that I had not been able to do much except registering on these platforms, wandering in a few test cycles without making myself useful, and all this to only earn a few euros. So I decided to continue a little more, and I finally spent a few months part-time.

Pros:

There is something positive about being a Crowd Tester, full time or just punctually.

• It brings you a supplement of income even a real salary according to your case, and especially according to your localization. Indeed, we are not equal on this planet and a euro/dollar does not have the same value in Western Europe as in Asia for example. And everyone is paid the same regardless of where you are living: London, Ukrainia or India.
• You can very well take this activity as a game: it can be challenging and motivating to play to find bugs and see the euros/dollars earned. If in addition you have the spirit of competition, because of rankings and various ratings obtained, there may even be a risk of dependence.
• It is a good way to learn, especially for a beginner having little experience of testing.
• You will discover many products and how to test these products. If you have never tested e-commerce applications or mobile applications, you will understand that this has some specificities
• You can see the tickets of other testers and progress to their reading
• You will have to force yourself to make perfect bug reports. Not like on your projects in geolocated teams where the developer is 2 meters from you and can come to see you for clarification
• These bug reports can be written in your language or in English (in general). Example: you have to know how to test an application in French and describe the bugs in English, or vice versa. It is always amazing (and weird) to have to describe a French spelling mistake in a bug report in English.
• You will have to use tools you did not know
• You will learn to work on very short deadlines, where the speed of writing is a strong asset. You have to be fast while being fair and precise

Cons:

But there are also many problems and difficulties:

• The need and the perimeter to be tested is always very difficult to apprehend. You do not always know the context of the project, and you will discover the product by using it. You will not always have a clear and precise understanding of what the customer really wants, and you can not question it, since you only have access to an intermediary who often does not have the answer. It is very frustrating to be so far away from the client and to feel so limited. You’ll do black box testing and will get almost no feedback on the work you provide.
• Many Crowd Testing platforms (but not all) use tools that are completely outdated and very limited in terms of functionality, making communication, understanding of need, and filing issues very painful.
• I have only experienced functional tests, and as a French resident, my conclusion is that you will not be very well paid. You should not rely on this income alone if you are not a security expert who will be invited to highly paid projects and if you are in a very taxed country (yes it is a job, please see with your local legislation how this paid work will be taxed) with high standard of living. For France, in relation to the time invested and the costs/charges that are charged to you, this job seems to let you earn less than the (SMIC) minimum wage even if you are very efficient.

The different services tested in detail:

Here are my impressions after spending several weeks as a Crowd Tester for these services.

Crowd App Quality — QAOnAir

I may have missed something in the process, because I did not receive any invitation for projects after registering.

UberTesters

To register, you had to answer a questionnaire, which really reminded me of the ISTQB questionnaires, and which, as you can imagine, put me off. I did it all while screeching at almost every question, selecting the least worst answer even though “it depends” would have been almost every time the best answer, sadly not available. I then received the Holy Grail: an email telling me that in case of a project, I would be kindly contacted. But I have not received any real Crowd Testing project; just an offer where I had to take the taxi as many times as possible in Paris with a mobile application, for an undisclosed payment. I may have missed the offer of my life because I did not answer.

• Financial gain: 0€
• Time invested: 2h

TesterWork

The invitations received by email expire very quickly, and the few times I managed to register, the confirmation arriving 6 hours later, it was already too late to expect anything from the test cycle. I also happened to immediately connect after being invited and realize that a hundred bugs had already been found, the first several hours before.

So I suspect this service to add testers later if needed, but since you need to be the first to find bugs to be paid, it becomes almost impossible to value your time spent if you’re not one of the first invited. In addition, some cycles come back, again and again, and the known bugs are filled in a huge spreadsheet with several tabs for each environment and OS. It’s like searching a needle in a haystack, titles being often very approximate.

I managed to quickly test only one application, found a crash, filled the bug and it was accepted, but whew! It feels like a 50% discount day in an Apple Store (only in your dreams). Note that TesterWork claims to have customers with big names like Spotify, Facebook, Evernote…

• Financial gain: $8
• Time invested: 6h

BugFinders

In order to participate, a demo application needs to be tested, and you must find 5 bugs to be part of the platform testers list. Once done, I received several project offers, and I tried several cycles. For some of them, it is necessary to use a VPN tunnel. The day a new version was released, some projects asked to keep using the old one which was not very convenient. In addition, the first password I used contained a ‘+’ which was not accepted and it bugs me.

There is no chat room, so it’s impossible to communicate with project managers or other testers who often have questions about the perimeter not always well described, or about an unclear feature. The website found itself several times in maintenance, without any warning sent to testers (at least I did not receive one). Not cool. But what is just not serious at all is to send an email saying that the bugs would be paid up to £10.00 with £2.00 bonus.

Finally we realize once registered that the minimum payment (which we receive in 90% of cases) is £0.5 and that the maximum payment of £10 is no longer mentioned at all.

I sent an e-mail to tell them how disappointed I was about this fact, and never received any answer: it was several months ago.

Nevertheless, I managed to find 2 bugs: one was accepted for a payment of £0.00 (not even a cent, thanks) because there were more or less similar problems on the same page, and the other one was refused. I communicated that I disagree with the fact that this bug was refused, arguing rather well, in my humble opinion, but I never had any answer too: it was several months ago.

The problem I had raised was on a mobile website. At the bottom of the page were links to Twitter, Medium, Facebook, YouTube social networks. I have all these applications installed on my mobile, and by clicking on the icon, the application is launched, except for Facebook which opens a new tab of the browser and therefore not the application. The bug was rejected because considered a suggestion, and the project did not accept suggestions. For me, it is clearly not a suggestion but a real inconsistency. And even if it’s not a big deal, not receiving any answer during the dispute made me run away and not coming back.

• Financial gain: £0.00
• Time invested: 5h

TesterIO

It seems to me that this offer is basically German with now a location in San Francisco. It’s a big player in the market, and a lot of projects are available: I’ve been invited a lot and spent a lot of time there. Some projects are sometimes very short (a few hours) and come back very often, probably with only a few modifications. I was even told that some critical bugs could be filled several times when the test manager did not have time to provide a good list of known issues.

I was not invited to cycles with test plan to execute, but apparently they sometime suggest this. TesterIO plays a lot on the scoring of testers, and a ranking is diffused and visible by all. Of course, the higher you rank, the more you’ll be invited to interesting and rewarding test cycles. It is therefore necessary to invest a lot before obtaining a substantial remuneration. I tried, but unfortunately I suffered several failures, some of which are difficult to accept, especially when you are an experienced tester.

For example, I did not like the fact that I could only add videos in mp4 format that can be opened in a browser. Their bug review tools are probably very limited and that’s why they require only one format, while any OS supports a lot more. What a waste of time, especially when in my case we use several Crowd Testing services, it is necessary to each time ask yourself which capture tool to use, or invest in a tool that allows all possible formats. Boring!

The requirements are often very unclear, it is very easy to be wrong and raise issues that are not valuable for the customer, customer with whom you have no contact. As for the known bugs, they are hard to navigate, and all the previous bugs rejected are not included. So, it is very easy to lose a lot of time on a bug when it would have been very simple, with some effective communication, to realize that it was an already known one.

Moreover, you know almost nothing about the project until you have accepted it, and finally realize that you will not be able to do it, or that it simply seems obvious to you that it will not be remunerative nor intellectually interesting.

There is a chat available for communication with the Test Team Leader, but it must have been developed in the 80’s. For example it is not even possible to attach a screenshot or a video, just plain text. Seriously?

You can be paid if you reproduce someone else’s bug: it’s a really good idea to rely on other testers to make sure the problems can be reproduced. But this is not well paid (0.5€, yes 50 cents) and not paid at all if the bug in question is refused, which will be communicated later. The payment can be made more than 2 months later, and it is necessary to request this payment in a hidden and poorly presented functionality in the interface. Yuck!

There is something that I had trouble understanding with testIO in the first place, it is the concept of Content Issue. As it often concerns e-commerce sites, the content (the database of products) is well separated from the whole business. And almost all test cycles consider content issues to be out of scope. It is not always easy to know what is considered content and what is not. It probably requires a lot of experience on the platform to be comfortable with this and know what is the content and what is not. Of course, you have no information about the product architecture.

Some examples of bad experiences with this service:

• I had videos refused in a ticket with a good detailed description of the steps to reproduce, being asked to record again the 5 videos in the proper format. Finally, my ticket was considered OOS (Out of scope). Did you first read it before asking me to record the videos again? Grrr!
• I had several other bugs rejected because out of scope, but it was just obvious that this perimeter was poorly described. I have tried several times to discuss this with the Test Team Leader. But it is clear that you have no power, and even if the error can be attributed to them, regarding the customer, they can not just accept bugs that are out-of-scope, because at the end the customer will refuse it. For example, I tried some search in the product with text encoded in Cyrillic, Arab or Czech. I found and filled issues, which was denied: but how should I have known that it did not interest the client to whom I can not speak?
• It is only possible to speak in the public chat and your comment will be read by everyone. Perfect to have a docile crowd 🙂
• When you do not agree with a denied bug, you have the option to “argue” by specifying it in a big google doc and then wait for an answer. The process is quite efficient and well-honed, but of course it is hard to make them change their mind, and above all it is irrevocable. Too bad for the customer if you found something that would be of great value to him. In addition, if you talk too much about bugs, you become banned and can not discuss anything. This happened to me, and it is hardly acceptable to say that the bugs that I found, and which were refused, can no more be discussed, challenged. It was at this point that I decided to leave the service.

This platform review was a long one, but it is true that it was with tester.io that I felt the most aggrieved and disappointed. I’d like to add that I was able to talk with someone who was top ranked and who makes a living from this activity, since the income he receives is more than enough for his country of residence. So we can be very satisfied too.

• Financial gain: 23€
• Time invested: several days

uTest

I do not have references, but it is probably the market leader, proposed by the company Applause. There are a multitude of projects to which I have been invited to participated in. And this is the one for which I spent the most of time. I can not say why I was more involved in the game of uTest than testIO, because I also had many disappointments. Projects are classified into several categories: Functional, Usability, Load, Security, Localization, Automation. In your profile, you must note your level of expertise on these different topics and therefore you will be invited according to this criterion, but also according to your country. I heard that there could be very few projects for some countries, whereas for me, in France, there were quite a few, only positioning myself on the functional, usability and localization. uTest often offers test cases to execute for a runtime payment and not only for bug founds.

To give an idea, I executed 104 test plans, reported 189 bugs (including 30 rejected) in 53 products tested.

A few comments:

• The requirements are often not clear enough
• The chat does not shine either by its effectiveness, one is also often disconnected, and there is no direct link between the conversations in the chat and the project page. No way to attach image or video file … Seriously? It is also disturbing to participate in a test cycle on one of the leading products of collaborative communication while using a tool worthy of a newbie.
• Communication is not always easy with TTL (Test Team Leader), especially because of the previous point, but also if he·she is in a different timezone, and if he·she does not master the project and the product, which happens from time to time.
• The interface is not convenient to use at all, but accepts different formats for capturing videos …
• The bugs are sometimes reviewed more than a month later, and the bi-monthly payments delay your payments even further.

It is almost possible to be busy full time with uTest.

• Financial gain: $2750
• Time invested: equivalent of 5 or 6 weeks full time

We Are Testers

Proposed by the company Stardust with its community We Are Testers it is a Franco-Canadian offer proposing missions for testers located in various countries. The latest missions were looking for testers from Belgium, the Netherlands, France, USA, Germany, Italy, etc.

There are not many projects proposed, so it’s difficult to get a significant income unlike uTest, but the projects are, in my opinion and in my experience, much better prepared and easy to integrate. You are warned a few days before, you can prepare your schedule accordingly, and because there are relatively few testers per cycle, not being active when the mission begins is not necessarily a problem. You can allocate time in the evening if you have a daily job and if the project is active during several days.

You can register with your Linkedin profile that will be reviewed, then with a kind of online review (I do not remember the content, but probably a multiple choice). I was initially refused understanding later that it was because I passed the exam too long after my Linkedin profile was accepted. I was lucky to find someone to talk to, and to finally be accepted.

A few comments:

• They created a Slack workspace and that, compared to the other offers, is a plus. The communication with the leaders of the test cycle is greatly improved. You can add a video/screenshot in the format you prefer, you can talk with other testers easily, you are notified…in short, you probably know what a tool like Slack can offer for a group working on the same tasks in terms of communication. Good point anyway.
• The interface is very well thought, nice, ergonomic and undoubtedly the most practical for the management of bugs. You can add any type of video and screenshot, and there are not 50 rules to respect. We feel like working with human beings who are able to understand a text with a demonstration video. In any case, it is a pleasure to use.
• The price per bug is (most of the time) clear: 6, 8 or 10 € depending on the severity of issues found. But I also worked on missions whose payment was poorly explained. Including a mission with a long document with steps to execute, and details in the interface was still mentioning the classic payment: 6, 8 or 10 € per bug. In the end, the 46 bugs that I entered were not paid at all, only the execution of the steps of the document. Not cool to learn it after spending 10 hours working hard. [Edit: An additional bonus was finally given when the team of WAT realized that it was not very fair. Good point]
• The questions I was able to ask had quick answers, even if they implied a customer feedback
• I did not have to discuss bug refused except the 46 bugs that was not paid in the first place (see above)
• You are paid very quickly, within a few days. The bugs are indeed reviewed soon after the end, and immediately paid.

• Financial gain: 552 €
• Time invested: equivalent of 2 weeks full-time

Personal conclusion

All services are not equal, and you will not be able to make a right choice by following the advice of a person. If you want to try Crowd Testing, you really have to invest a lot of time before you can really make a profit (maybe less than the minimum wage depending on the country you’re located). If you are looking for an additional income, I can say that in my case, I lost a lot of time and hairs during the first month, before understanding better what was expected or not; Only after spending at least 1 month on the same platform, may you pretend mastering the entire process and earning a little money.

Then you have to know how to stay faithful. In my case, being unemployed during a few months, I wanted to test the Crowd Testing a few weeks. I finally took the game and continued for several months until I found a full-time mission, leaving me with no more time to continue. The writing of this article also marks the end of this experiment, I will not do any more test campaign with these platforms except in case of another inactivity period, and only with those with whom it was easiest to work, probably uTest and WAT. But do not expect, if you are in a similar situation as me as a freelance paying all taxes in France, to earn more than the minimum wage with these platforms. Better to serve Big Macs.

Originally published at www.lyontesting.fr.