Computer Vision: Magic Wand Annotation Tool

For supervised machine learning it is necessary to have precisely labelled data. Annotating data is a tedious and cumbersome task, especially when it comes to a massive amount of items to label with a complex structure respectively border.

For example, I want a segmentation network for dogs, there are different kinds of dogs and for every frame they will have a new pose.

Figure 1: Source: the PASCAL VOC project

To make things easier and gain speed while annotating we created a magic wand annotation tool which helps you finding complex borders of an object just by setting a rectangle-like region of interest (ROI).

Implementation

For the magic wand we compare the pixels in the image in terms of similarity, thus we can group similar pixels by spectral clustering and create a contour of the object.

There are some questions that arise when comparing pixels, one can compare pixels by:

• RGB features
• Light intensity
• Texture
• Positional coordinates
• Hue

In addition, if you compare every pixel with every other pixel a massive operator is created, e.g. for a ROI of 300x300 pixels, one creates a 90,000x90,000 similarity matrix, a symmetric matrix which describes the similarity of every pixel to every other, which requires a considerable amount of computation resources and memory.

So we have to decide which features do we compare and how to deal with that amount of data.

After some research we ended up by using a combination of RGB values and coordinates and implemented an approximation scheme of the large similarity matrix, which reduces the amount of data to be processed, so we can compute the cluster in real time. In addition we use a self learning similarity metric.

Figure 2: from left to right, original image, drawn ROI, computed contour of cluster

But it is still tough for the computer to select the right object with the most homogeneous part, since it can’t guess the purpose of a human. Plus, for an object with a complex pattern in itself, it is hard to get some similarity.

To summarise, the magic wand has its limits but works fine for objects which are quite homogeneous in itself but with a complex contour.

In the end, the magic wand annotation tool is not magic but pure math.

You can try the magic wand annotation tool yourself, just visit our MoonVision platform and start a labelling job, for free.

Outlook

We need to go deeper! Currently we are investigating a second wand-like annotation tool which uses a more semantic constraint but pure RGB data, a deep embedding space, where you also partially define what you are interested in. So stay tuned for my next update!