Be the Boss — Mechanical Turk Assignment
M-Turk Assignment — Tag 50 images.
Questions for the assignment -
Briefly describe your HIT design. What did you do to encourage high-quality inputs? What did you change from your initial prototype to your final one?
I had divided my images into 2 different tasks -
The first one had 20 images of animals of my recent trip to the zoo (with 2 exceptions). The users were supposed to identify the type of the animal in the image, describe the activity its doing and also write down a line of thought about what the animal is potentially thinking.
An example of a HIT in the zoo tagging task can be seen below —
The second task (I tried for 30 remaining images ) was essentially giving a tag to the image and estimating the price of the product in the image. These images were taken from various sources online to test out assignment.
An example of the Price Tagging task in Mturk -
My initial prototype was one whole survey that included all the images on a google form. However, i realized it would be very unlikely for workers to complete the single survey in a huge task with high quality..
So I used the Mturk Image tagging template that divides each image into various HITs (like micro-tasks) and provides it to the worker. The worker will get paid to tag each image. To encourage better output, I also made sure they get paid slightly more than usual image tagging tasks on mturk. (I had also tried limiting the workers with masters qualification. But the output was really low with it).
What do you think about the data you received from workers?
The data i received was quite interesting and accurate actually. People did make a lot of mistakes in spelling (I guess due to language concerns), however most of them tagged the images quite accurately. When it came to a creative task, the amount of data was quite varied. Some tried to answer with really creative funny quotes and others were just explaining rationally the situation. I wonder whether changing the time-limit per HIT would make any difference.
How much did it cost you to complete your HITs? Where would you estimate the hourly rate?
It costed me around 6$. However, the amount of time spend on each HIT was pretty less. An hourly rate could easily have gone upto 10$ for the tasks I had posted.
One thing I noticed was how easy the experience of Mechanical Turk was as a Requester compared to being the Worker. As a requester everything (including the Payment obviously) was pretty smooth with quite less usability issues. The website felt different as a requester.
The google doc with actual tags and results are compiled here -> Be a Boss Result.