Socially Responsible Data Labeling
Generating a global training dataset while supporting social initiatives and sustainable practices
Labeling satellite imagery is the process of applying tags to scenes to provide context or confirm information. These labeled training datasets form the basis for machine learning (ML) algorithms. The labeling undertaking (in many cases) requires humans to meticulously and manually assign captions to the data, allowing the model to learn patterns and estimate them for other observations.
For a wide range of Earth observation applications, training data labels can be generated by annotating satellite imagery. Images can be classified to identify the entire image as a class (e.g., water body) or for specific objects within the satellite image. However, annotation tasks can only identify features observable in the imagery. For example, with Sentinel-2 imagery at the 10-meter spatial resolution, one cannot detect the more detailed features of interest, such as crop types but would be able to distinguish large croplands from other land cover classes.
Human error in labeling is inevitable and results in uncertainties and errors in the final label. As a result, it’s best practice to examine images multiple times and then assign a majority or consensus label. In general, significant human resources and financial investment is needed to annotate imagery at large scales.
In 2018, we identified the need for a geographically diverse land cover classification training dataset that required human annotation and validation of labels. We proposed to Schmidt Futures a project to generate such a dataset to advance land cover classification globally. In this blog post, we discuss what we’ve learned developing LandCoverNet, including the keys to generating good quality labels in a socially responsible manner.
LandCoverNet: A Global Land Cover Classification Training Dataset
LandCoverNet is the first global land cover classification training dataset based on Sentinel-2 imagery that (will) contain labels from all continents and account for labeling errors. In an expert workshop, we gathered ML and land cover domain experts to define the specifics of this dataset, and one of the recommendations was the inclusion of label uncertainty in the final dataset.
Sentinel-2 imagery has a 10m spatial resolution, and some land cover classes are hard to identify at this resolution resulting in a human judgment error. With the advice of community experts and the success of a consensus algorithm developed by our collaborator Prof. Lyndon Estes, we decided to label each image chip three times with independent users. This translates to a labor-intensive labeling campaign that requires a large and dedicated team to label the data. Our goal: ensuring such a campaign is carried out with a social impact, and responsibly. That’s how we got to know the TaQadam team.
TaQadam is a social enterprise focusing on creating a digital economy in the post-conflict affected areas, employing primarily displaced and host communities. Their team is fully remote, working on mobile and web softwares from home in a group and/or team setup.
TaQadam’s team has worked on many geospatial labeling projects, including image segmentation, object detection, and image classification. While our tasks were image segmentation, the use of Sentinel-2 with the 10m spatial resolution was new to them.
TaQadam gathered a team of 35 annotators coming from different backgrounds, including Syrian, and Syrian Palestinians alongside different communities in Lebanon itself. They are graduates from digital economy training of Acted, World Food Programme, and Digital Opportunity Trust. They are mostly high school, trade school (vocational digital course is considered one), or university students living in Beirut and some areas outside the capital who are looking for such opportunities outside full-time employment.
Radiant’s team created tutorials and documentation to teach the group how to interpret land cover classes from Sentinel-2 imagery. We also hosted online training for them on using the annotation dashboard. Following this, we conducted a pilot campaign to familiarize the TaQadam team with the data and various challenges that may arise.
First Release of LandCoverNet
LandCoverNet V1.0 which covers the African continent was released in summer 2020. For this version, the TaQadam team labeled more than 380,000 image chips with 7 land cover classes. The resulting dataset has close to 130M pixel labels and consensus scores.
Throughout the campaign, we also labeled a set of images in our team as “expert labels.” Using these expert labels, we assessed the accuracy of labels from annotators, and provided them an average accuracy score to improve their decision-making for selecting specific classes. Examples of challenging classes were also shared with them as use cases to learn about the properties of various land cover types. This approach helped address human errors in the annotation process, resulting in quality labels that accurately capture and map diverse global land cover types.
Next, LandCoverNet South America
We are currently working with the annotators to label the chips in South America. While smaller than Africa, it still requires labeling 230,000 chips. This dataset is planned for release in summer 2021. With the support from our cooperative agreement with NASA Earth Science Data Systems (ESDS), we will expand the dataset to be a fusion of Sentinel-2, Sentinel-1, and Landsat 8 with labels derived from Sentinel-2.
With the impact of COVID-19, and lockdowns around the world, the TaQadam team has continued to work as the team is entirely remote. We look forward to continuing our partnership with them and expanding the Radiant MLHub data catalog.