How to rank & review your annotators

Some useful tips and suggestions that may be useful for anyone who is managing a team of annotators

Sep 27 · 4 min read

By Uxío García Andrade(👐Open Link to this article)

In our latest update, we have included a new 🆕 feature to our search engine, which allows users to filter documents based on whether a given user has confirmed them (or any user has confirmed them). This feature, despite it may not look like such a big deal, it can be used for many different things. In this post I will discuss some of the applications 🚀.

Your annotators are working hard. Help them a little.

First, let’s see how it is possible to use the query in the tagtog web application. You just need to navigate to a project (e.g. Then input this query in the search bar located at the top left:


where username is the name of the user you want to know which documents has completed. If you use * (“star”) instead of an username, you will get the documents any user has completed ✅.

Searching in the GUI

You can also use logical operators, such as OR, AND or NOT to make more complex queries. Here you can see an example of a query which retrieves the documents that have been completed by the users demo or uxio:

Complex query example

And now, we will get to the first of the applications I mentioned before. Let’s review step by step a script that allows you to get a report 📘 of how many documents any member of a given project have completed.

Before doing anything else, I define some variables that will be useful later on:

Constant definitions of your project

First, I get a list of project members, who have annotation permissions, so I exclude those users who just have the role reader assigned:

Get list of annotating members

Then, I have written this function, which uses the search engine API to retrieve the documents completed by a given user:

Getting the number of confirmed documents

Finally, we just need to output the results in a practical way. I have chosen to print the results to a txt file, but it could be any other format 📜.

Printing the results in a file

Below is the output. As you can imagine, this allows you to periodically track and rank which member is doing more annotations:

Output example

Moreover, another possible application of this feature would be to get a list of documents which have been annotated by any user 📝. This is very useful for those members who review the annotations of others.

For this, we will use a similar function to the previous one, but we will use * instead of an username:

Get all the documents completed by anyone

Afterwards, I iterate over the list of documents I have obtained in the response, and I print the document ids in a different txt file:

Printing the results to a file

And again, this would be the output:

Output example

Curious to apply similar searches to your project? Click here to obtain this post’s full script’s code. Moreover, we have other code samples in our github repository:

I hope you found this new search feature useful! Let us know in the comments your experience 👇🏻.

Needing training data for #NLP? Find it, create it, and share it on: 🍃tagtog

Like birds? Follow us on Twitter 🐦

👏👏👏 Clap if you like the post, and want to share it with others! 💚🧡


Written by

The text annotation tool to train #AI. Easy. 🔗

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade