Figure8 Tutorial

Panagiotis Mavr
May 7, 2018 · Unlisted
Tutorial for FigureEight platform

In this tutorial we will learn how to use Figure8 platform (formerly known as Crowdflower). More specifically we will learn the following:

Contents

A. Prerequisites

B. Using FigureEight interface to create and execute tasks

C. A glimpse of CML for customised tasks

D. A glimpse of the Figure8 API on how to create and edit a task (Python)

E. Credits

F. Resources

A. Prerequisites

Before moving on we should make sure that some systems and accounts are well set.

Jupyter notebook

We will make our Figure8 api calls though Python so make sure you have installed a Python 2/3 compatible version of Jupyter notebook. You can find an installation here with relevant instructions. You can use pip to install jupyter in python.

Github account

Get a Github account for free. I highly recommend you the Github student pack.

Figure8 account

You can get a free trial or a free academic/student account for Figure8 with the following ways:

  1. Free trial
  2. Github Student Pack (not available anymore as part of Github Student Pack)

Python libraries

Use pip to install the requests library so that we will be able to make http/https requests to the FigureEight API.


B. Main tutorial

  1. Design a task from scratch with data
  2. Select the participants that will answer to our tasks
  3. Reassure some quality for the answers using gold/test questions
  4. Visualise Answers from participants
  5. Get the results report

Task design

if logged in at FigureEight: Now press the create job and you will be redirected to the previous picture.

Screen from jobs view in your account

Then if you press on the “Create Job”. (..or If you are new to FigureEight starting a task you will see this screen):

Choose from templates or make a new job from scratch

A very classic task where humans are better than machines is the character recognition. We will start with a transcription task taking some captcha images and ask the participants to recognise the text from the captchas.

Even though we actually are doing text transcription we will go at the bottom of the page and press, start from scratch.

A typical captcha image
Several captchas

Insertion of Data

Then we will be redirected to a page that will look like this where we will be asked to insert some data:

We will use this custom data “FigureEightTurorialData.csv” file. Download the file from the link and drop the data in the box.

Then a screen similar to this will appear:

If our data appears correctly then we are ready to go to the next step which is the design of the actual task.

Designing Custom Questions

We will have the following screen which constists of the title, the content and the instructions:

Design Task page

There we can choose a title for our job. Let’s name it “Captcha Task”

Content

Then there is two parts for the Content: Data and Question.

Data appearance

We have already uploaded our Data. Now we should show how the task data will appear. Our captchas will be nice to appear as links or images. So let’s make them appear as links and images to facilitate the workers (we will do the image with CML to facilitate it more).

If we click in the data area we0.3 have the option Add Data:

We press the Add Data, then choose the only column that we have and add them as image.

After we insert the data as image and links the screen will look like this:

Let’s add some questions for our Task.

We will add questions to:

  1. ask for the captcha characters. We click on Add Question and select the single line Text:

2. ask for the readability of each captcha image.

As you can see both questions are required.

Instructions

This part is very important. Depending on the task we could insert an overview, steps rules or tips and specific examples that could help the worker understand the task.

Preview for task

Before going further it will be nice to take a look at the task and check that it will appear correctly to the workers.

Quality

After we insert these questions we can create test questions if we wish. FigureEight might complain about it at this point but we will be able to insert them at a later step if not now.

If we press into creating test questions a menu like this will appear:

Test question editing for a particular row

Since this is a task that we have launched before it is interesting to see the distribution of answers and the propositions for the answers that we get. This can facilitate a lot the task design and testing.

If we add two questions, this is how the quality view will look:

2 test questions enabled for a particular task

In the settings menu we can perform several controls. We can select the level of participants (for external channels), we can perform an internal channel job (free of charge as we have to find a crowd for that), specific geographic location etc.

Selecting participants

Participants can be selected according to geographic location, level, channel, etc.

“Geographic location”

Geographic location: We can select participants if needed according to geographic location. We can safely assume that people living in a country will know more about it. We have the ability to target specific countries with FigureEight.

Levels: 0, 1, 2, 3. When you start at FigureEight your level is 0, unless you have completed enough tasks you don’t get to level 1. Then you advance with your progress towards level 3. The level 3 participants are the more tested participants. They are significantly less and this has an effect on task completion.

Channels

FigureEight has different channels were they get the workers.

Quality Control

Under Quality tab there is several options. We select the Dynamic judgements since we have already set up some test questions.

We selected in purpose 3 judgements per row as maximum for dynamic judgements and we set the confidence on a result to be at least 0.7. You can see that we selected only the character fields as field with dynamic judgements.

Pay

If we use an external channel then workers need to be paid. We can select how much they will be paid depending on the judgements and the price per row or price per page. You can see the pay settings in this screenshot:

Payment settings for workers

Task execution

If all the above steps have been done correctly we can launch our task!

Pressing on the launch tab:

This is an example of launching of a task similar to the previous one with the 4 rows of data that we had but with no Test Questions created. We can see that we get a warning at launch time so that we take this into account. We can see before we launch that we have an estimate for the total cost of the task and that we need to provide with 10 dollars as a buffer to the platform and we will get the funds back only when the task finishes.

Monitoring

At this point we have already launched and finished the task!

We can see that we got successfully 12 judgements as we asked and we have 2 active test questions. We can spot the trusted and untrusted judgements along with other characteristics.

Also we can spot the satisfaction of participants which is fair given the task design and instructions we gave them.

Distributions of answers

Another important characteristic is the visualisation of different answers we get from participants. We can see how their answers are grouped into percentages if we press the “Advanced Analytics” tab and then select the “Distributions” from the drop down menu.

A screenshot of our distribution of answers for the captcha characters:

Distribution of different answers for our captchas

And for the readability:

Results


C. Custom CML task

We will take a look at the cml of the task we just created. CML stands for Custom Markup Language and is an abstaction to go from text to interface design in FigureEight. This will be very useful for our next step which will be designing a task with the API calls through Python.

<div class=”html-element-wrapper”>
<a href=”{{link}}” target=”_blank”>
<span>
<img span=”” class=”fr-fic fr-dii” />
</span>
<span>{{link}}</span>
</a>
<p>
<span>
<img class=”liquid-image” src=”{{link}}” />
</span>
<br />
</p>
<p>
<br />
</p>
</div>
<cml:text label=”Please enter the characters of the Captcha” validates=”required” name=”please_enter_the_character_of_the_captcha” aggregation=”all” gold=”true” />
<cml:ratings label=”What was the quality of the readability of the captcha image?” to=”Very Readable” from=”Very Unreadable” validates=”required” name=”what_was_the_quality_of_the_readability_of_the_captcha_image” aggregation=”avg” gold=”true” />

Html part

There is a part that wraps the html elements and spans all the data we have provided and presents both links and images for that.

CML elements part

Also then there is a part where we have the two different fields per captcha:

  1. <cml:text label> gets as input a single line text

We can see the different attributes that it accepts and that gold is set as true.

2. <cml:ratings label> gets as input one of the 4 labels that we set from 1 to 4

We can also notice that is has the gold option enabled.


…If you want more about CML you can look it up at the FigureEight CML tutorial at their site.


D. FigureEight Api

In the following we will create the same task as before but with the use of HTML for instructions, CML for the main task using the FigureEight API.

The code used for this Tutorial is available in Github. Be aware as this code could updated.

The API key

Before making any call to the API you need to have the API key to be sent for authorization purposes.

Where to find the API key?

Step1

Account details

Step2

API key will appear on your left.

Giving others your API key will give them access also to the funds that you have in the account as a person or as a team. You can generate new API Keys at in order to prevent API key leakage.

Python Libraries to use

We will use the “requests” library and the “json” library for Python.

import requests
import json

2. Creating a blank task

api_key={randoms_string_given_by_FigureEight}

response=requests.post(“https://api.figure-eight.com/v1/jobs.json?key="+api_key)

This will create a blank task with 0 rows, no data, no title, no task questions and no instructions.You can test it by yourself.

Response code

#response: status code
print(response.status_code)

Unless this value is 200 there is a problem with this step that we need to check. A documentation of the different responses can be found here.

Response JSON object

We have requested from the URL:

https://api.figure-eight.com/v1/jobs.json

This url as a response gives a JSON file with details about the job we just created.

Let’s print the response and see what we get.

#response: content
print(response.content)

Example of JSON print

b'{"id":1264679,"options":{"logical_aggregation":true,"mail_to":"p.mavridis@tudelft.nl","req_ttl_in_seconds":1800},"title":null,"secret":"yIeDQfT0GE0uWJBEObi7mS8yLuC4rvCzqOAE4FIBVDAH","project_number":null,"alias":null,"judgments_per_unit":3,"units_per_assignment":5,"pages_per_assignment":1,"max_judgments_per_worker":null,"gold_per_assignment":1,"minimum_account_age_seconds":null,"execution_mode":"worker_ui_remix","payment_cents":10,"design_verified":true,"public_data":true,"variable_judgments_mode":"none","max_judgments_per_unit":null,"expected_judgments_per_unit":null,"min_unit_confidence":null,"units_remain_finalized":null,"auto_order_timeout":null,"auto_order_threshold":4,"completed_at":null,"state":"unordered","auto_order":false,"webhook_uri":null,"send_judgments_webhook":null,"language":"en","minimum_requirements":null,"desired_requirements":null,"max_work_per_network":null,"copied_from":null,"created_at":"2018-05-08T19:30:37+00:00","updated_at":"2018-05-08T19:30:37+00:00","included_countries":[],"excluded_countries":[],"instructions":"","cml":null,"js":null,"css":null,"confidence_fields":null,"gold":{},"units_count":0,"golds_count":0,"judgments_count":0,"support_email":"p.mavridis@tudelft.nl","worker_ui_remix":true,"crowd_costs":0.0,"quiz_mode_enabled":false,"completed":false,"fields":null,"order_approved":false}'

3. Create a task with title and instructions and cml (no data)

We will create now a task with title and instructions.

First keep the title and the instructions into a variable.

job_title = “Captcha API task”

instructions = “<h1> Instructions </h1> Here Give some useful instructions for the participant”

cml=”””<div class=”html-element-wrapper”>
<a target=”_blank” href=”{{link}}”>
<span>
<img span=”” class=”fr-fic fr-dii” />
</span>
<span>{{link}}</span>
</a>
<p>
<span>
<img class=”liquid-image” src=”{{link}}” />
</span>
<br />
</p>
<p>
<br />
</p>
</div>
<cml:text label=”Please enter the characters of the Captcha” validates=”required” name=”please_enter_the_character_of_the_captcha” aggregation=”all” gold=”true” />”””

#setup the request url

request_url = “https://api.figure-eight.com/v1/jobs.json"
headers = {‘content-type’: ‘application/json’}

#Setup the payload object

payload = {
‘key’: API_KEY,
‘job’:{
‘title’: job_title,
‘instructions’: instructions,

‘cml’: cml
}
}

response=requests.post(request_url, data=json.dumps(payload), headers=headers)
print(response.content)

4. Add to a task data (row by row)

To perform this we will need the job_id. The job_id is in the response.content object. We need to parse the JSON data to get it:

#find job_id and save it
workJson = json.loads(response.content)
job_id = workJson[‘id’]
print(job_id)

Then we can make can make the request to add row by row data on the task:

#Updating the task we had created before by adding data row by row (unit by unit)
API_KEY = “jgEAbFKSCLtee6sYz59N”

data = {“link”: “https://www.scienceabc.com/wp-content/uploads/2016/07/Captcha-ex.jpg"}

request_url = “https://api.figure-eight.com/v1/jobs/{}/units.json".format(job_id)
headers = {‘content-type’: ‘application/json’}

payload = {
‘key’: API_KEY,
‘unit’: {
‘data’: data
}
}
response=requests.post(request_url, data=json.dumps(payload), headers=headers)

This will add the first row of the data. Ideally, we could parse an object and add the different rows automatically or add the rows manually.

#row per row posting of data

print(response.content)

data = {“link”: “https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTA-Gi8p3udDewYGuYnAh2ai3J-c29APFuGBJpmMSt7hIVmAesqSQ"}
payload = {
‘key’: API_KEY,
‘unit’: {
‘data’: data
}
}
response=requests.post(request_url, data=json.dumps(payload), headers=headers)

Overview of API for FigureEight

We learnt how to use the api to create a task. We can also update a task title, update the cml, update or add data, launch the task etc.

To make a FigureEight API call we need:

  1. to know which Url to use with which parameters
  2. a payload object (with the api key and other parameters information)
  3. set the headers to json or csv depending on the case
  4. to know what kind of request you make is post, get, put
  5. depending on the request response you might need more.

For more about the different requests that can be done in the FigureEight API you can take a look at:

  1. Different requests on FigureEight
  2. API integration of FigureEight
  3. FigureEight API at a Glance site.

E. Credits

I would like to thank Dr Alessandro Bozzon that trusted me into providing TU-Delft Human Computation Course this tutorial on FigureEight. Also I would like to thank my friend Dr Pavel Kucherbaev that inspired me by sharing his previous tutorial on FigureEight. Finally, I would like to thank my colleagues and friends Dr Andrea Mauri and MSc Carlo Van Der Valk for helping me out with their precious Python skills.

F. Resources

  1. IPython notebook source code on Github
  2. FigureEight platform API
  3. Different requests on FigureEight API
  4. API integration of FigureEight
  5. Insomnia: API Call “Translations”
  6. FigureEight CML Reference
  7. Python API Tutorial
  8. FigureEight Data (rows)

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade