Bossing around for a day
An exercise in crowdsourcing on Mechanical Turk
As I searched for pictures from my archive, I realized that I had snapped many flowers while being oblivious of their common names, so that became my first task to crowdsource! Since these were flowers from the Indian subcontinent, I expected some guessing (and possibly, dishonesty) due to lack of knowledge from turkers on the other side of the globe and hence was constantly monitoring to block anyone doing labeling poorly — surprisingly, I didn’t notice any attempts to game that were blatantly apparent. Note that as I didn’t know most names myself, I relied on agreement among the responses, which worked fine as a strategy in this case with few conflicts!
Next, as I had only 32 unique flower pics from one particular trip, I decided to randomly pick from images I had collected over the years and have them analyzed for potentially interesting tweets. I expected some to be hard to come up with, this, to get better responses, I casually announced the possibility of a reward for exceptional work. A flaw I observed was that many other HITs with the keyword “tweet” in some form involved actually tweeting, so I rephrased the title to use the word phrase instead of tweet to avoid confusion.
Flower-naming being a knowledge-dependent task, it was paid better initially at 10¢/tag, with more time allotted too for workers to check their work from other sources. But the assumption was proven false as it was taken up immediately, with 5 unique workers attempting it by the time I wrote this. The other task with a social aspect didn’t find takers oddly enough, so I bumped the rate from 7¢ to 10¢. Since the interface didn’t allow me to modify a batch of HITs once requested, I didn’t lower the rate for the more popular task. In entirety, this exercise is expected to cost about $19.5 including a fairly substantial $4.5 in site fees.
The data for both tasks along with outputs (as comments):
- 32 Flowers at goo.gl/photos/AkPLnqo9x4DTyN47A
- 18 Tweets at goo.gl/photos/8fnnqCfdngXutoZB7
To reiterate, I’m quite ignorant about nomenclature of flora, so I relied on agreement (and lack of collusion) among those who would do the first task. As of now, almost 80% of the pictures have been tagged, with about 75% of them identified unanimously. I was happy to note the lack of any typographical errors and that many turkers seemed to take more time to verify their answers. Also, some took intermittent breaks after doing a few HITs at a time. Once I have all the data, it might be interesting to calculate the actual duration of work per HIT on average. As for the tweets, I will be offering a bonus to the super-enthusiastic participant who came up with 36 pretty good tweets, including one in apparently inept Spanish (though still managing to get the point across).
I will be updating this post as soon as the tasks are complete or the HITs expire (Wednesday afternoon), to provide the final cost evaluation. The estimate, which AMT takes as credit, comes to about $8.33/hour for the flower-naming task and $7.8/hour for the tweets: for flower-naming, I planned about 45 seconds to come up with an educated guess and 25% overhead, requiring 30 minutes for each of the three workers & for the tweets, I expected about 60 seconds of effort, coming to 18 minutes for the whole set.
