Testing Areas and the Training Workflow

Hello fellow Picterrans!

On February 10th, 2020 we released a brand new feature that is now an integral part of your workflow when training a detector. This feature is called Testing Areas. The purpose of Testing Areas is to define areas where you want to visually assess the results of your current training iteration so you can decide where your detector needs improvement in the next training iteration (after clicking ‘Improve Detector’). But before we get into the details, let’s provide some context.

We now have Training Areas, Testing Areas and Accuracy Areas. You might be a bit overwhelmed, but don’t worry, we’ve got our experts working away on a snazzy solution to help you breeze through the learning curve! But before that, we have this blog post :).

Let’s do a bit of review on the tools available in the toolbar:

The training toolbar

From top to bottom,

• Results: Simply hides and shows the results card used to change the results display as well as show accuracy scores generated from Accuracy Areas. The functionality already existed, we just moved it to this button.
• Select: Let’s you select and edit existing areas and object outlines.
• Training Areas: There’s a lot more info on this in our previous post here, but in short this defines what regions of the image your detector will use to learn what is and isn’t your object of interest.
• Testing Areas: The new feature to be discussed below!
• Object: The drawing tool that let’s you do the outlining/annotating of your objects of interest within your Training/Accuracy areas. You can choose either an arbitrary Polygon or Circle tool.
• Accuracy Areas: Generates an accuracy score for your detector. The tool allows you to define regions that you have to fill with your object outlines (just as in training areas). Unlike Training Areas, Accuracy Areas are not used in training the model. The detector will generate predictions within Accuracy Areas, compare those results to your object outlines and generate a set of scores for your detector. The more (in both quantity and diversity) Accuracy Areas you have the more representative your accuracy score will be of your detector.
• Import: If you already have a geojson that defines areas or objects you can import it here with this tool.

So what are Testing Areas?

Testing areas are where your detector will output results after you click the ‘Train Detector’ button. The purpose of Testing Areas is to visually assess the results of your detector while you are still in the process of building the detector. Therefore you do not need to outline the objects inside of them as you normally would with Training/Accuracy Areas.

Testing areas will become a core (and indeed required) step of your workflow when training your detector:

• Step 1: Define Training Areas
• Step 2: Draw Object outlines within those Training Areas
• Step 3: Define Testing Areas

Previously you could only assess your training results on a single image at a time. Now with Testing areas you can control exactly where your model outputs results during training. You can draw them on multiple images and your detector will generate results across all testing areas in a single run!

This does 2 things for you:

• It lets you quickly grasp where your detector needs improvement across a multitude of potentially very visually different images, facilitating your workflow in creating much more generalizable and accurate detectors.
• Previously if you trained on a single very large image it could take over 10 minutes since it needed to generate detections across the entire image. Testing areas allow you to speed up your training iterations by giving you the freedom the focus detection generation on the parts of the image set that are representative of the whole set.

This is what they look like:

In Red, testing areas. In Yellow, training areas. Training results (not to be confused with Detection results) have only been generated within the testing areas.

We have some more helpful tips on how to effectively utilize testing areas in our custom detector crash intro!

How does this fit into my overall workflow?

Another major reason behind Testing Areas is that we want to make the distinction between Training mode and Detection mode clear.

Training Mode:

• Build your detector with Training Areas and Object outlines
• Visually assess its performance using Testing Areas
• Improve and iterate on your detector by adding and adjusting (training/testing) areas and object outlines at each round of training.

Detection Mode:

• Run your trained detector at scale and in parallel across your entire set of images to generate a complete set of results.

To push this notion forwards every training run will now be charged at a low flat rate independent of the size of your testing areas! This is so that you do not feel as constrained while you are training your detector, allowing you to improve it up until the point you are satisfied with its performance. Then when you want to see and export final actionable results across your entire image(s), you will need to run your trained detector in detection mode.

Cheers and happy training!