Labeling Guides and Exams: new Supervisely tools to teach workforce on your custom annotation requirements

Creating big and diverse computer vision datasets requires working with a huge labeling workforce. That, in turn, requires professional tools to educate and examine them.

Why do we need Labeling Guides and Labeling Exams? Let’s consider several important observations:

• usually, an annotator has concrete domain specialization: self-driving cars, agriculture, satellite imagery, medicine, and so on
• not every annotator properly understands labeling requirements, thus sometimes they make systematic errors during labeling
• it is a good idea to choose for your custom task the most efficient and accurate labelers based on Quality Score
• from labelers perspective, comprehensive annotation policy should be presented as easy to learn guides and materials
• from time to time company changes annotation policy, all labelers have to be notified and re-examined

Labeling Guides and Labeling Exams in Supervisely are designed to handle all these aspects.

Labeling Guides —videos, documents, markdown posts are all in one place

Labeling Guides page

Labeling Exams + Quality Score— measure how labelers understand task

A manager sees the table with exams and their general information.

Labeling Exams page

The row can be expanded and the internal table with all labelers' attempts will be displayed. That table contains Quality Scores and links to Exam Reports.

Expended table with Quality scores for every attempt

This view allows to see the big picture and monitor the entire examination process. Also, it helps to dig into details and see how every labeler performs.

Exam Report — detailed error analysis at your fingertips

There are several tables with geometry score, objects count score and tags score to calculate more complete Quality Score. Also, there is a report table for every image with a bunch of statistics and calculated visual differences.

Detailed Exam Report

Notifications for managers and reviewers

Notifications allow you to don’t miss relevant information and stay informed of what’s going on.

Notifications pop-up

Can I try it now?

Sure! It’s already there — select Labeling at scale -> Exams in the left menu and the new tool will open. If you don’t have an account in Supervisely yet — sign up to Community Edition for free now.

If you found this post interesting, then let’s help others too. More people will see it if you give it some 👏.