One year of Custom Detector
It has been almost a year since we released the first version of our Custom detector tool. This is the story of how it came to life, how it improved and what we learnt along the way.
The 3 bets
One and half year ago, when we started brainstorming ways to increase the number of detectors on our platform, we came up with 3 options:
- Add additional built-in detectors that we train on large datasets (SpaceNet, XView, …)
- Allow users to sub-categorize the output of built-in detectors (e.g. detect houses and then classify each house into having a solar panel or not)
- Allow users to build their own Custom Detector: A Deep Learning model that is trained to detect whatever they are interested in
The first two options were things that we knew how to do: we had done it before and that’s the process we were used to as ML practitioners.
The third option seemed a bit crazy: we had no idea how to do it and we had no idea if it would actually work: most of the ML literature would make you think it wouldn’t.
But we thought that the bet was worth making and that it would be too bad to discard option 3 without even trying. We could also always fallback to one of the first two options if things went wrong. The Custom Detector was born.
What’s in a Custom Detector ?
A Custom Detector is a Machine Learning model that detect objects in images. To build such a model, you need:
- An image
- Some example of what you want to detect (training annotations)
- An area in the image where all the examples are annotated (training area)
Our vision for the first version of the Custom Detector was really to automate something that the users would have done manually otherwise. For example, counting crops in a drone image of a farm field. We also wanted the Custom Detector to work with few examples — let’s say 10 annotations — and be an iterative process. So training a detector should take in the order of minutes and the UI should be quick & intuitive.
We started some tests with some popular cutting edge detection models, but we quickly figured out that:
- Training them is more in the hours/days ballpark than minutes
- They require a large amount of annotations
This is because these models are complex and were designed for complex datasets like MS COCO which are multi-class, multi-scale problems.
Our focus is on geospatial imagery and one of the defining trait of them is that they are taken from above and have a known spatial resolution (e.g. one pixel is 0.3cm by 0.3cm on the ground). This means that you don’t have objects that appear smaller because of their distance. You basically don’t have to deal with perspective.
This and the fact that we went for single-class Custom Detector allowed us to start with smaller models. Our first Custom Detector model took about 10 minutes to train on a K80 GPU.
Ship a simple model and then iterate
This initial version of the Custom Detector was honestly embarrassing. The engineering team coined the concept of ‘Maximally Embarrassing Product’. Although it did work on some use cases, it failed pretty badly on a lot of others. We also knew that this was a first version and had a long list of ideas to improve the model.
Yet, we shipped it and we learnt so much…
First, we learnt by looking at what kind of objects users were trying to detect. We had talked to people before and therefore had some ideas of what they would be interested in, but once we made the Custom Detector available to anybody, we got people trying to do completely unexpected use cases. This ranged from detecting wooden debris in rivers to cattle counting.
As embarrassing as it was, this early release of the Custom Detector allowed us to start aggregating the various datasets that users were building, which bootstrapped our model improvement efforts.
Our experiment framework
After releasing this first version of the Custom Detector, we shifted our effort to build what we call our “experiment platform”. This is a collection of tools that allow us to run a new version of the Custom Detector model against a collection of users datasets and see if it improves. This is an invaluable tool for two reasons:
- By automating most of the process of testing a new model, we were able to quickly implement, test and integrate/discard new ideas for the Custom Detector
- By relying on user datasets instead of traditional dataset (e.g. SpaceNet, various challenges), we ended up actually tweaking for what our users were doing instead of what everybody is doing
After a few months, we ended up having what we believe is one of the largest dataset of remote sensing cattle imagery.
In the end, 6 months after the release of the Custom Detector, with the help of our experiment platform, we drastically increased the accuracy of the model.
This means that by pooling and aggregating the data that our users upload and generate, we end up creating a better Custom Detector and all of our users benefit from this. So each additional user brings some marginal improvement to the Picterra model.
With these continuous improvements over a year, we now have a robust Custom Detector that is flexible enough to address use cases ranging from object counting to semantic segmentation.
Takeaways & the future
One takeaway of this story is that when developing Machine Learning based software, releasing early is as important as for any other kind of software. It will allow you to learn about your users and make sure that your improvements are going in the right direction.
The other takeaway is that by aggregating the various user datasets we obtained by releasing the Custom Detector early, we were able to improve it much quicker than if we would have to collect this data ourselves. So this is the occasion to thank our users for this and reassure them that even more improvements will come in 2020 thanks to them using us !
Now that we have these solid Machine Learning foundations, our focus for 2020 is on scaling detectors to large geographical areas. We’ll do that by focusing on two axes:
- Shipping a brand new UI to allow detecting over large areas or a large number of images
- Continuing on improving our API to allow users to integrate Picterra in their own solutions
