8 min readFeb 27, 2019

This is the second in a series of blogs where we will write about some of the problems we are trying to solve at Playment and how we’re doing our bit to advance the AI age. You can read the rest of them here: https://medium.com/playment

Computer vision systems today rely heavily on deep learning algorithms. These models in turn, rely heavily on training data. By feeding in appropriate training data, computer vision systems can be developed to find relationships, detect patterns, understand complex problems and take decisions.

Better the training data, the better the computer vision system performs. Quality, Variety and Quantity of training data determine the success of the system.

To learn more about Playment, and how we help companies and startups fulfil their training data needs, read part 1 of our blog here:

Generating training datasets at scale is a challenging problem. It requires a lot of operational overhead and gruelling quality checks. All in all, a lot of time and effort for our clients.

This is where Playment comes in. Our suite of annotation tools help AI companies to accelerate the development and testing of their computer vision models.

Playment’s suite of Annotation Tools.

Our annotation tools are designed to build high quality labeled datasets for machine learning models. AI assistance, superior UX design and configurability makes our tools smarter.

Our tools support data formats from various sensors. They are architected to facilitate fast feature deployment for custom data management.

Various annotation types supported by Playment.

Platform to build training data is incredible complicated to design and develop. It involves complex workflows and complicated systems which enable accumulation, visualisation, cleaning, labeling, sampling and quality checks of the training data.

Annotation tools are very integral part of the training data platform.

As computer vision systems get more ubiquitous, the need for such tools will be even more important.

In particular, we’ve built up a vast amount of tooling that assists humans in writing 1.0 code, such as powerful IDEs with features like syntax highlighting, debuggers, profilers, go to def, git integration, etc. In the 2.0 stack, the programming is done by accumulating, massaging and cleaning datasets. For example, when the network fails in some hard or rare cases, we do not fix those predictions by writing code, but by including more labeled examples of those cases. Who is going to develop the first Software 2.0 IDEs, which help with all of the workflows in accumulating, visualising, cleaning, labeling, and sourcing datasets?
– Andrej Karpathy (Director of AI, Tesla Motors)

Data in → Annotation Platform → Data Out

On the most fundamental level, our annotation platform takes in raw data, allow annotators annotate the raw data, and finally generates high quality training data.

The raw data can be in different formats, based on the different sensors (cameras / LiDARs / RADARs) and their configuration in the computer vision system.

The final annotated data can also be in different forms: semantic segmentation, polygons, poly-lines, 3D cuboids, etc for various use-cases. Detailed list of all the annotation types supported by Playment can be found here:

Who uses these tools?

Building these tools is very challenging. One of the most challenging part is that the annotation tools are used by lots of different types of people in different ways.

Some personas we’re building our tools for:

Computer Vision Engineers: the engineers who work on the computer vision systems.
AI Product Owner: the product managers who build AI powered products. They also own the business and product KPIs.
Operations: the operations team at Playment responsible for ensuring that the data labeling happens as per the needs of the customers, and with appropriate quality benchmarks (precision & recall), within the turnaround-time, and within acceptable margins.
Annotators: We have two kinds of annotators: the in-house annotators for projects with stringent security requirements and a managed community of over 30,000 well-trained part-time annotators from smaller cities in India.

What’s even more interesting is that all these different personas use the annotation tools in different ways, and hence have very different needs and priorities. For example:

For an Annotator: speed and efficiency of using the annotation tool might be very important.

For a Computer Vision Engineer: the quality of annotation will be of paramount importance. The ability to explore the dataset, identify edge-cases, systemic errors would be important.

At Playment, we’ve researched long and hard about these different personas, and how to build our tools to serve them all. As a result, we’ve identified four basic use-cases which we try to fulfil:

Visualize: being able to visualize the data, and dissect it across different dimensions and features.

Annotate: annotating the data with very high accuracy and efficiency. Sometimes we also fast-track the annotation process by having the AI models work alongside humans by auto-suggesting, or checking their work.

Quality Check: sampling the dataset – and then identifying systemic errors in the dataset.

Manage: setting up the data workflows, to ensure that that annotations happen within the appropriate quality, cost and time benchmarks.

Product Principles

These are some of the product principles we keep in mind while designing and developing our annotation tools:

Configurability and Modularity

Most of our customers have very specific and nuanced annotation requirements. This is because the business context for most of these startups and companies is very distinct.

Let’s take an example – for a self-driving truck built for inter-city transport, identifying lane markings and road signs is very important. On the other hand – for an industrial robot, identifying nuts, bolts and other tools becomes more important.

As a result, we’ve made our tool modular and configurable — so as to cater to diverse and nuanced annotation needs of our customers.

Hardcore focus on Accuracy

Deep learning models need very high degree of accuracy to function correctly. This means that all our annotation tools are designed keeping that in mind. Advanced quality check features, automated heuristics checks and other features help us increase the quality of the datasets.

Focussing on Efficiency

Over the course of a month, an annotator on Playment would’ve worked for over 200 hours, annotating 1000s of frames, drawing some 100,000 annotations.

10% more efficiency in the annotation tools directly leads to 10% more time saved for the annotator, which in turn helps them be more productive and earn more money on our platform.

Optimising for every interaction, every keystroke becomes super critical at the scale we operate in. A slightly modified flow, a few more hotkeys, or having some power features usually go a long way to help our annotators.

Ease of Use and Onboarding

Most of our annotators are not familiar with such complicated tools. We take immense pride in being an online platform where anybody can sign up, learn, and start earning. However, this means that we have to make our interfaces intuitive enough to be used by anybody and everybody.

Imagine teaching somebody the basics of Photoshop, or After Effects, or from scratch — remotely — and ensuring that they reach 98%+ accuracy throughout.

Technical and Design challenges

Should completely run on the web.
Should be easy enough for 1000s of annotators to learn and use.
Should run on computers with hardware limitations.
Should not consume a lot of network bandwidth.
Should be modular and configurable enough to support future use-cases.

Playment’s Annotation Suite

Despite all the technical and design challenges, we have built an amazing suite of annotation tools. They are being used by dozens of enterprises to accelerate their AI development process.

Shown below are some of the tools and tools we’ve built at Playment:

Semantic segmentation

Product to build pixel-perfect semantic segmentation training data at scale.

3D Point cloud annotation tool

Visualising, labeling and tracking objects across frames in 3D point clouds for all types of LiDAR data.

2D Shape annotations

For object detection and localization in images and videos.

Advanced Quality Check Tools

For computer vision systems to work perfectly, the training data fed into the models needs to be of very high quality. This becomes especially important when these systems are used for autonomous vehicles (where a single error might lead to an accident or loss of life).

Hence, we’ve built tools and features which help to calculate the objective quality metrics and margin oferrors for the datasets. We push our products to be proactive rather than reactive in identifying systemic errors which might creep into the datasets.

Release / Post Release

Timeline of release

It took us about a year to build out the entire suite of our annotation tools. Countless hours were spent in designing, developing, refining and iterating on them.

Reception to our Products and Tools

We’ve work across the board with AI software startups, ride-sharing companies, automotive OEMs, large automobile manufacturers, etc. In the autonomous vehicles industry, we have a lot of companies and startups who vouch for the quality of products and services provided by Playment.

We involve our customers actively in the product development process, and use their feedback and suggestions to iterate and improve our products.

All this has been possible only because of the hard work, and efforts of Akshay, Sarthak, Prerak, Abhilash, Satbir, Maneesh, and the entire Product, Technology and Design team at Playment.
Together, we’ve spent thousands of hours designing, prototyping, coding, refactoring, testing, and taking feedback from our customers and annotators.

If you find these problems interesting, and are looking for new challenges — we’d love to have you here at Playment. You can know more about the open positions at Playment at https://playment.io/jobs/