Integrating a Qualtrics Survey with Amazon Mechanical Turk

There are multiple ways to recruit uncompensated volunteers for your research. For any case, you just need to send the link to your survey, and the participant will do the rest. You can hope to get at least an 8–10% response rate on your survey. In most instances, there will be some incentive for the participant to complete the survey, such as a sweepstakes or monetary reward. For the latter, it is in your best interest to keep track of your funds and the quality of the responses you receive while still protecting privacy and anonymity. By integrating Amazon Mechanical Turk and Qualtrics, we can easily compensate survey takers by providing your survey link to identified participants (region, age, language, etc) and easily transfer funds after they complete their task successfully.

It is important to keep in mind that Amazon Mechanical Turk workers vary in intention: while some workers participate out of interest, other workers may complete surveys as quickly as possible, skewing accuracy. You should always build safeguards on your survey (Forced Validation and attention checks) to make sure that the responses are accurate.

Survey Design

Let us first go over some basics regarding how to divide the survey by sections. This separation allows us to implement the forced validations and attention checks easily, as well as facilitates the location of the mTurk validation code, which connects a particular mTurk worker with a given survey.

First, locate the survey that you are going to link with mTurk. If you have paid and unpaid versions of the same survey, it’s a good idea to create two versions and label them properly.

When you click on your survey, you will see all of your questions (image below) on the same block. It’s good practice to divide your survey by sections. The division will also make it easier to identify key parts of your survey as well as the right location for your validation questions. Select the questions that you want to group and click the Move to a New Block option on the right of the selected questions.

Survey Custom End Message

You can change the name of blocks according to the type of question. Click on the Library tab on the far left of your screen. Then, select “Message Library”.

In the Message Library section, you will see all of the messages available for your surveys.

Click on Create a New Message in the Message Library section. Now select End of Survey Messages.

Enter a short description for your message and write your customized end message content.

This {e://Field/mTurk} Qualtrics code will show the unique identification code for the survey taker at the end of the survey. The field name mTurk is linked to the flow of the survey. You will link the message and flow later during the creation of the survey flow.

Survey Flow

The Survey Flow allows you to customize where participants go and what they see. This feature gives you a quick overview of your survey and how the participant navigates through your survey. Here you can check the logic of your survey and implement the forced validations and attention check actions. In the main survey window, select Survey Flow.

In the main survey window, select Survey Flow. You will see your questions in blocks in the same order they appear in your survey.

At the end of the flow, there is an option to add a new element. Click Add A New Element Here and select Web Service.

In the URL field you can use any random generated number URL, or you can use a Qualtrics URL. Keep in mind that some URLs may change, so always check that the URL is returning a random number:

http://reporting.qualtrics.com/projects/randomNumGen.php

Click on Add A Parameter To Send To Web Service and enter the parameters. In this case, the parameters are random numbers in a given range. For the purposes of this example, we will use the range 1 to 99999999.

Because we want a random number to be generated at the beginning of each individual survey, you will need to drag and drop the Web Service block created in the last step to the top of the flow.

In the Set Embedded Data field, enter a name for your field in the left hand box and enter “random” in the right hand box.

Proceed to Create A New Element; however, this time, select End of Survey. Click customize. In the Customize End Of Survey box, select Override Survey Options, and change Default End of Survey Message to Custom End of Survey Message.

From the drop down menu, select My Library: Your Name and then select the custom end message you created in a previous step.

This is an example of a completed Survey Flow.

Amazon Mechanical Turk (mTurk)

In this section, you will learn how to create a survey link request to hire Amazon Mechanical Turk workers. mTurk workers come from a vast pool of talent and from a diverse array of backgrounds. Keep in mind, you will want to insure your survey workers fit the demographic sought after in your research.

Log in to your Amazon account (you should create a new account for mTurk purposes). On the homepage, select the create tab then select New Project.

Select Survey Link from the toolbar on the lefthand size of your screen.

Follow the instructions to create a new project. Enter project name, description, and keywords for your project.

Read the instructions in the Setting Up Your HIT section. This is where you determine how many workers your project requires, payment, and length of research.

Click Advanced and click No. By selecting No, your research can be accessed by any mTurk worker, rather than only by Master’s level workers.From the drop down menu, select Customize Worker Requirements. Now, you will be able to specify all qualifications workers must meet. Select the qualifications according to your needs.

In this case, we changed Master to selected Location (USA).

We also need to change the HITs approved and approval rate to a lower one so we can attract new Amazon Turk users. A message about not selecting Master Workers will be prompted. Click OK.

Here, you can customize your survey link message. This is what the Amazon workers will read when they opt to take your survey. There is also a section for workers to enter the unique number that they will see at the end of the survey.

Click Preview. This will save your work. Now you can see how your survey link will look for workers.

After you preview your message, click save. You will be taken back to the main window. Now you are ready to publish a batch of your survey. A preview HIT section will show a summary of your request.

You will have a chance to change the Batch Name and Description and review your request.

Finally you will be prompted with a breakdown of the cost of your request and how many funds you have available.

Conclusion

At this point, you should have successfully created a survey with the correct settings that you can link to your Amazon Turk account. You are now ready to receive responses. You can expect responses to come in rather rapidly. Remember to review the speed with which workers respond. It is your decision whether or not you want to disqualify workers who complete the survey in too little time to have actually read the questions. This is where your attention marks and forced validation will play a vital role in data accuracy and proper distribution of funds.

Reference

jose Luis Rodriguez

Written by

Director of Margolis Market Information Lab at University of Illinois at Urbana-Champaign. go.illinois.edu/marketlab