Crafting the future was just the beginning
On the 14th of March 2018, I was proud to share some significant semantic segmentation progress in an article entitled “Machine Learning Serving the Industry.”
On the 14th of September, Airware¹ officially shut down its business. An event nobody expected. Since then, against all the odds, we have been able to secure the product and continue the adventure! You will find more details hereafter.
I wrote most of this article while the shut down was still fresh to me, before moving into the closing process. I wanted to share what we were about to unveil to the world as part of our product: automatic digitization of mines and quarries. Something nobody else in our industry has shared as of today.
The closing process became unexpected in its form and content, and we fought hard to lead it into an overwhelming conclusion! Therefore, I completed that initial writing with some insights of the past weeks and some events which were gratifying for Redbird and in particular my team.
This article covers one of our most significant achievements, something I cannot be prouder of. It emphasizes some of our biggest strengths and differentiators while not diving much into technical details.
This article is a part of my professional life as a tech manager. It is accessible to a wide audience. You will enjoy it even more if you have a certain taste for technology.
Introduction
Redbird wanted to become a hub for the data coming from the physical world. A platform to aggregate data from our customers. Airware later shared its vision: becoming a digital twin of the physical world.
Airware was selling business insights and a geographical business intelligence platform that helped our customer take more reliable decisions and faster. Mainly in the mining and quarry industries, along with construction and some Insurance.
In the Airware organization, I was the Director of Engineering leading the Paris office engineering and data-science team. As former CTO of Redbird — acquired in 2016 by Airware — I was also providing guidance regarding the platform and product development.
Everything we were selling was distributed through the platform we initiated at Redbird. We have a lot of friendly customers testimonials describing how we were changing the way they work.
The following screenshot highlights automatically digitized stockpiles as our customer could see them in our front-end (internal code name: Cardinal²).
The digitization process leads to several different automatically identified “objects” or classes: water, vegetation, construction, vehicles, haul roads, stockpiles, safety blocks, safety berms, quarry faces.
This process — involving machine learning — takes less than 20 minutes for our largest covered locations: hundreds of hectares / hundred of millions of pixels in multiple dimensions. That duration includes the machine learning inference, the digitization — transformation of pixels into polygons, lines, and points — and the data publication on the platform.
As a result, our customers can access CAD files (like DXF or SHP) along with interactive in-platform editable objects built for collaboration: annotations³.
For a Human, that process would have taken several hours for stockpiles only, and several days for all the classes. Here comes our objective: reducing as much as possible any manual activities, down to 0 if that is achievable.
When I lastly introduced this new capability to one of our customers in early September, I first had to convince him it was real and then the reaction was tremendous:
What you built is incredible. You are going to redefine the business. I want it now!
For a purpose! His company — one of the world top 10 in the construction, mines and quarries industry — has an entire entity in charge of manual digitization; tens of people are working 5 days a week to manually identify classes as mentioned earlier.
This feature available in production was about to redefine the day to day job of all the people in that entity, reducing their manual digitization process by 95% to 99%; enabling those Human experts to work on other tasks with a better added value.
All the analytics we were selling were based on those digitizations. We had traditional algorithms performing as good as they could, and requiring much manual effort to match our quality standard. We have successfully solved that Human requirement, replacing the traditional algorithms by advanced machine learning.
How is this tour de force possible?
This achievement is the combination of the most advanced machine learning models and technics operated at scale through cutting-edge serverless architectures leveraging cloud computing and homemade crafted frameworks.
Machine learning
Semantic segmentation is the practice of categorizing an image at the pixel level. i.e., semantic segmentation assigns a class to each pixel in the image.
Detecting the bird’s presence in the image is not enough. Semantic segmentation also delineates the boundaries of recognized objects. Unlike classification algorithms, semantic segmentation operates at the pixel level.
When we want to delineate stockpiles from roads, water from vegetation, vegetation from buildings, and so on. We are describing semantic segmentation.
There are several semantic segmentation challenges like Pascal VOC, COCO or Cityscape.
As of today, computer vision has progressed to the point where Deep Neural Network (DNN) models have become widely admitted as top performing solutions for most recognition tasks such as classification or segmentation.
In that context, it may seem straightforward to solve our problematic: “digitization of the physical world from the sky.” Not at all!
We have faced a wide variety of challenges —impacting all aspects of Machine Learning — to make that happen:
- Iron constraints
On 29th of September 2016, Amazon Web Services announced P2 instances types. One week earlier I was negotiating the price of a dedicated machine with NVIDIA Tesla K80 graphic cards to support the first training we were about to launch as a Proof of Concept to validate the feasibility of the project we then called GanNet⁴. I was lucky enough, so the responsible for purchase order was out in personal time off. The week after, I canceled the quote for the dedicated machine! We have never stopped performing Deep Learning training in the cloud since that time. At our peak, we had up to 20 p3.8xlarge instances training several models versions for several weeks in a row. Without a cloud provider such as AWS, we wouldn’t have been able to craft our model and train GanNet that fast. - Unique training dataset
None of the existing contest datasets represent drone data and resulting products such as what we were producing: drone images stitched together thanks to photogrammetry into large GB orthophographies and digital elevation models and then processed to become valuable analytics. Fortunately, we have started to acquire data and produce analytics back in 2013, and we have assembled so much dataset and analytics for several years serving numerous customers that we had plenty of data for training. Therefore we step into the training database completion — to make it compatible with supervised learning and for all requested classes, inputs and expected outputs. No secret here, the solution was business expertise, brain juice, elbow grease, and patience. - Specific Model architecture
Top leading pre-trained models and architectures were performing poorly for our industry and our data. Transfer learning technics has provided poor results — results not sufficient to prevent manual edition of digitized polygons, lines, and points. One of the main reason was the poorness of their receptive field — we are trying to segment object larger (in pixel) than the largest images of any of the challenge for which on the shelves models were conceived. Not to mention that contrary to the vast majority of public datasets, the model has to process GB of data for each inference (Pascal VOC entire database is a couple of GB only).
Overall, it took us several months to prepare and validate the training and validation database. Moreover, it took us several months to craft and train the model that performs the best. From the idea to the golive, it has been a two-year long project.
We have imagined and implemented several “secret” ingredients to make it work: from the input logic to the architecture of the network itself.
In a few words, we have designed a deep learning architecture tied to geospatial data — exploiting both colors and elevations at different resolutions. Our model is acting much like what the Human vision does when looking at a scene: trying to accommodate and see details at different scales to finally build the overall understanding.
Nevertheless, semantic segmentation by itself is not enough to get digitization of asset. Consecutive and correlated colored pixels must be transformed into polygons, lines, and points intelligently.
Here, intervene more “traditional methods,” business knowledge and rock-solid computer science to get from pixels to polygons as fast as possible.
There is so much more to share on that topic. I won’t unveil everything in a row. 😊
Buzzard
buzzard is a fantastic python open source library that we built to ease the manipulation of raster and vector files at a large scale.
Manipulating GIS files is not a native task in python. A 3rd party library is necessary for any operation. Essentially, 3 open-source choices were offered:
Python bindings for GDAL and OGR:
- All features are available for both raster and vector files
- They are an abstraction built on top of 10 years old C++ code
- Unpleasant documentation
- An OGR wrapper
- A poor design not serving what we needed
- A GDAL wrapper
- Has almost no tool to manipulate sub-rectangle of rasters
Since no good solution was available we decided to create our own open-source library: buzzard.
buzzard is the synthesis from several years of experience manipulating geospatial data at scale. The initial requirements:
- Raster and vector files opening,
- Raster and vector files creation,
- Raster files reading to numpy.ndarray (a N-dimensional array used across data science project in python coming from NumPy),
- Raster files writing from numpy.ndarray,
- Vector files reading to shapely objects, geojson and raw coordinates,
- Vector files writing from shapely objects, geojson and raw coordinates,
- Tools for rasters windows manipulations,
- Tools to ease the manipulation of rasters and geometries together,
- On-the-fly spatial reference reprojection between opened files like a GIS software.
buzzard has supported all our python projects since it’s ready. It has proven itself better in many ways than other libraries.
buzzard is already in use in other companies, I foresee a bright future for it!
Therefore, we still plan to spend some time on it for the sake of seeing it last.
Data analysis framework
Once our Machine Learning algorithm is ready for prime time, integrating it into our platform requires:
- An infrastructure to operate it at scale
- Tools to connect it with the rest of the platform
Therefore we built the data analysis framework. It is both an abstraction layer for our data scientists algorithms that enables and simplifies all interactions with the platform and a powerful tool that enables running algorithms at scale in production.
The data analysis framework has been designed for:
- Extensibility
Crafted to run any type of algorithm either it is CPU or GPU based. It leverages docker images. - Efficient development
The SDK handles all APIs call, the data-scientist focus on the business. - Scalability
The framework runs on serverless infrastructure, and it is virtually infinitely scalable. - Rapid prototyping
Once a data scientist has understood the SDK, deploying a new algorithm is effortless. - Efficient processing cost
As it is based on serverless infrastructure, the operating cost is incredibly low.
Serverless architecture
The data analysis framework described above leverages Serverless services provided by Amazon Web Services.
Serverless doesn’t mean there is no server, it means that a tier is taking care of all server aspects for you. Typically, a cloud provider — in our case AWS— operates the servers, dynamically managing resources allocation. The pricing is based on the actual amount of resources consumed by your application, rather than on pre-purchased units of capacity. Often, Serverless services come with higher level features like an orchestration API.
Our architecture:
Consumer Layer
The Consumer Layer is the entry point to launch the workflow. It leverages AWS API Gateway (1) to trigger every data analysis. It is a new endpoint added to our API.
Orchestration Layer
The Orchestration Layer integrates all analysis services from the computation layer to automate and sequence all steps of the data analysis process. It allows isolating the workflow definition from the execution of the individual jobs. In this layer:
- AWS Lambda (2) plays the trigger’s role. It gathers any additional required parameters and launches AWS Step functions (3).
- AWS Step functions (3) is the orchestrator. It defines and orchestrates the data flow. Step functions is a state machine that defines a series of steps and passes appropriate information from one step to another.
- AWS Lambda (4) also executes simple tests or action related to orchestration: perform test conditions, submit jobs to AWS Batch (5) on the computation layer, check jobs status of the computation layer.
AWS Step functions (3) also leverages shared microservices like the “Notification microservice,” “Development error microservice,” “Annotation API,” and “Document API.”
Computation Layer
Computation Layer hosts all data analysis services that are called by the Orchestration Layer. AWS Batch (5) allows to submit and run individual data analysis tasks with the ideal CPUs, GPUs and memory resources allocated. AWS Batch (5) executes two activities:
- Transferring data between AWS S3 (6) and AWS EFS (7): purpose is to copy orthophotographies and DSM to support high speed read and write for Batch.
- Executing the actual data analysis which can be machine learning inference or more traditional algorithms. Examples of algorithms running: GanNet, advanced contour lines, automated DTM, drainage analysis, etc.
Data Layer
Data Layer represents where data are stored, they are different data sources involved in the data analysis framework:
- AWS S3 (6) stores raster files (ie. DSM and orthophotographies) coming from the automated processing or manual upload from our production team/customers.
- AWS EFS (7) is a scalable, elastic file storage. It acts as a hot storage for the computation layer. It stores a duplication of rasters from S3. AWS Batch (5) effectively perform computations based on EFS through an nfs connection. It would be too slow to download GB of files every time a computation is performed.
Results
The following screenshots and animated GIF illustrate the results as our customers are able to see them on the platform.
Combined with our advanced volume computation method able to “clean”⁵ conveyor belts or trees on top of stockpiles along with determining breaklines automatically, we have the most advanced and fastest solution for mines and quarries production management and inventory.
Combined with our automated analytics generation, we have a fully automatic solution.
What else?
Since the announcement it has been incredibly crazy: from delightful customers and partners feedback to acquisition interests from competitors and other major industry players — an endless list of meetings, emails, and documents I have attended, lead and written.
The following highlights are some of the most interesting, surprising, unexpected and gratifying events.
Customer data management
We were an online service, a SaaS company. We were managing hundreds of millions of images for hundreds of Terabytes.
After the shutdown, we proposed our customers to download their data. It was the minimum to do after years of trust. Not an easy move. Thanks to our robust Cardinal platform, we have been able to implement a backup feature in a couple of days.
Still, some customer needed more than 1 Terabytes hard drives. Thank you mail services! 😅
Competition aggressive moves
- Few hours after the announcement, one of our competitors publicly declared — through different means including an official web page — that they would cover our contracts for free up to 2020. Since then, some of our customers started to process their data with that provider.
- The truth arises: our data has proven to be more accurate, we were providing correct elevation and correct local coordinate system, our automated detection outperformed that competitor’s (less than 5% of automatically detected stockpiles on their side compared to more than 90% with our technology on the same datasets), our customer support has been recognized as much more tremendous to theirs...
- Some of our customers even requested support from my team. Somehow annoying, partially embarrassing, surely satisfying.
Technical due diligence
- I have spent the 6 weeks following the announcement speaking with all our competitors interested in buying the Redbird team — Paris office. Either it was just the engineering team or the whole team. There were 50 of them from world leaders large enterprises to top startups all around the world.
- I learned a lot during this roadshow — that would deserve a whole article by its own. What was both surprising and pleasant was the way these competitors looked at what we built in the Paris engineering team. This technical due diligence granted me accesses to several places. Some of our competitors were jokes, and some truly impressed me.
- My team and our product were more advanced in several areas: understanding of our industry, verticals, applied machine learning, geospatial manipulation. With the potential buyers, we have imagined exciting industrial projects for our product and team. Several I would have been proud of being part of. One of this project concluded, product and team are pursuing the journey. This is a very exciting news!
Machine Learning challenges
- One of the highlights of the technical due diligence was a Machine Learning challenges we were submitted to by one of our potential “buyers.”
- This challenge aimed at processing never seen datasets coming from the potential “buyer” database through our automated digitization pipeline — as depicted in the above paragraphs.
- It took us less than an hour to process automated digitization (as polygons, lines, and points), automatic DTM and advanced contour lines (with smoothing and preserved topography) for 3 new datasets. The results were remarkable impressive and the feedback we received was amazing. This led to an offer 💪.
Conclusion
I couldn’t be prouder of what we accomplished. We did a fantastic job of pushing the boundaries, defining the field of possibilities. We have opened new doors and possibles.
Crafting the future was just the beginning, and with this week announcement, it is a work in progress again. The future we built was bright, and it still is!
I’m proud that Delair is pursuing the Redbird journey hiring the team and keeping the product alive. A new chapter has just opened!
“Innovate or die” has been in the Redbird team DNA since day 1. We have chosen: Innovate 💪
You may also want to read our entrepreneur journey in an article from Emmanuel de Maistre a Redbird co-founder.
Acknowledgement
Thank you Buddy, for being so supportive in the past years. Thank you Isaac and Pete, for being so enthusiastic regarding the data-analysis framework and everything it was enabling.
Thank you Dave, for trusting my team and me despite the distance and some misconceptions you have been given.
Thank you, my beloved team, for building all that with so much passion: Alice, Amaury, Christophe, Clément D., Clément F., Hawa, Jieren, Kévin, Niall, Nicolas, Olivier, Théo, Tristan. You are the best! Thank you to those who already moved onto new adventures.
Thank you Emmanuel N., Benjamin, and Emmanuel de M. for that tremendous opportunity you gave me while I was a suit up consultant.
Thank you to all my other fellow coworkers in Paris and San Francisco.
Thank you Lise, my partner in life, for all your helpful support, time, comprehension and help.
Thank you all for these remarkable years and those to come!
Notes:
¹ Airware official one paragraph press introduction:
Airware provides secure and scalable solutions that enable enterprises to digitize their business and turn aerial data into actionable business intelligence. World-class enterprises look to Airware to improve productivity, operational efficiency, and worker safety. Airware has raised more than $90 million to date from leading venture capital investors including Andreessen Horowitz, Kleiner Perkins Caufield and Byers, and Next World Capital, with strategic investments from enterprise customers and partners including Caterpillar Ventures, GE Ventures, Google Ventures, and Intel Capital. Airware’s Americas headquarters are located in San Francisco, CA and EMEA headquarters in Paris, France.
² Cardinal was the front-end serving all data we produced to our customers. From traditional base data: slope map, orthophotography, dynamic surface model to advanced analytics like safety insights hall roads width, faces heights, stockpile volumes, etc.
³ Annotations was one of the customers’ favorite feature. It was enabling collaboration, work tracking, notification, export, computation, etc.
⁴ GanNet, was the codename of the automated digitization project. It is a bird name as all our microservices. Our mono repository was named nest and we had several birdies. A commitment to our angry birds tradition 😊. Some other examples: automolus, koepckeae alias cupcake, mesange, seagull, parrot, logrunner, etc.
⁵ We were providing a set of methods for volume computations — covering all possible use cases. One of the method, publicly called “breakline” was capable of way more. One of its function was to erase unwanted data to provide more accurate results.
