What happened when we gave 13000 work assignments to anonymous internet strangers
We are creating Taskopus - a platform where you can give some strangers the instructions along with some data and expect the work to be done.
The twist is that there is no initial approval of the workers. You can just sign up. Nobody asks for your passport, nationality or address. Not even for payment information. Nobody approves you. You are a perfect stranger.
Another stranger can give you a task and some cryptocurrency for that (Bitcoin Cash (Why?, even if most cryptocurrency freelancers accept Bitcoin)).
As expected, people started to abuse the system from day one.
We expected people to try to create multiple accounts to avoid “per person” limits on tasks. We expected people to be randomly clicking to earn their money as fast as possible.
We even had people come out and tell us how they cheated, though it’s a totally different story.
What we didn’t know was how much should we trust our workers. How many of them would be doing the work well? 90% of workers? 50%? 10%? 0?
We have decided to test that.
The answer was a surprise for us.
What kind of work people are usually paid for?
It’s either the work that is very creative and few people can do it well (think: writers, artists, designers, architects, lawyers) or it’s boring and just needs to be done.
Taskopus is not very suited for the first kind of work, because it requires constant contact between a buyer and a worker. But it’s very well suited for the second type (boring, repetitive, but needs to be done).
The “typos” experiment
So, we have created this spreadsheet:
We call it “the typos experiment”.
So, what exactly is this?
This is a Google Sheet that serves as a basis for this task:
Here is a close up of a Google Sheet:
Each row of the Google Sheet will be a separate task for the worker (we call it the “table-based” projects).
As you can see — there are a few columns — “word1a”, “word1b”. One of those is the correct spelling of a random word, one is misspelt. Misspelling can be either at “a” or at “b” spot:
That way the worker doesn’t know where the correct word is and has to think. (The worker won’t see the “correct” column, more on that later)
Each typo is a random change of a letter.
We can now create a task template in Taskopus:
Each row from Google Sheet will be substituted and the resulting task will be given to worker.
We will give up to a maximum of 50 such tasks per worker (so, 250 words). Each task would cost $0.01 in Bitcoin Cash.
Then we will analyse — were the workers clicking randomly or were they actually doing the job. We will do that by comparing the worker’s answer to the “correct” column in the sheet.
In total we have generated 130 rows and will give each to 20 different users.
130 rows * 20 times each row = 2600 total tasks.
2600 tasks with 5 typos each = 13000 typos to check.
At $0.01 per task that comes to $26 for the experiment.
2600 tasks / 50 tasks per person = We can test 52 people or more.
Ok, let’s run it.
So, it generally takes people 14 seconds to look through 5 typos and advance to the next task (that is the median time of last 10 completions) and it works out to about $2.51/hour.
That means that in order to successfully complete the task the worker has to be attentive for about 12 minutes (50 tasks * 14 seconds).
The project is complete
A few hours later the project is complete and here are the results.
This is the output directly from Taskopus (in CSV format, uploaded to Google Sheets). It shows the user_id, the input data, and the answers of the user.
Now, with a little help from formulas we can compare the “correct” column to the answer of the user.
Here we have the columns with “1” for a correct answer and “0” for an incorrect one.
Let’s calculate the percentage of correct answers by adding the “AverageCorrect” column. That would help us create the pivot table.
Turns out that on average these random unfiltered users did remarkably well.
Nobody was there to control them and the tasks were being paid automatically.
Still, most of the users did an honest job for $2.70/hour
The numbers close to 50% are marked red here because it means that the user was randomly clicking answer A or answer B. That would make him right on average 50% times (due to random luck).
Let’s plot the users:
We have also stretched the data from [50%-100%] to [0%-100%], so “0%” here means “random clicking”.
As you can see a lot of people did the job quite well.
Results
The number of correct answers from all workers? 95.1%
Remember, those are completely anonymous random strangers doing the work for people they don’t even know.
21 out of 68 people (30%) did a “100% correct job”.
If we put a threshold at “98% correct” that would give us 49 workers (72%).
So, 72% of random internet strangers did a really good job.
The best part of Taskopus is that you can mark those people and give the next assignments exclusively to them. That’s called “multi-stage projects”.
By the way, you can run such experiments and request work like that yourself on Taskopus. It doesn’t require any special access.
Got your own idea to test? Try Taskopus!
