The role of AI in unlocking the potential of imagery for global insights
Originally published on the Taylor Geospatial Engine blog site on May 23, 2024.
As we have written recently, Taylor Geospatial Engine (TGE) created the Innovation Bridge framework to accelerate the commercialization of geospatial research. Our first Innovation Bridge initiative aims to expand and mature the market for applications of AI and ML systems on earth observation imagery. We believe this will help unlock the innovation potential of geospatial data in understanding global food security and sustainability.
Satellite data is especially useful for observing phenomena in agriculture due to the proliferation of sensors and sensor types in orbit. Globally-collected data provides comprehensive coverage on frequent timescales, at high resolutions, and in multiple spectral bands. This is crucial for many detections relevant to agricultural change. These detections are highly useful inputs for determining changes in agricultural systems. The demand to better understand our global food systems has helped fuel the rapidly expanding interest in applying AI and machine learning models to the growing body of available satellite imagery.
AI capabilities that are currently available commercially to recognize and predict features in imagery were generally trained on images of real world objects that would be captured in photographs, like cats. They were not designed for the patterns and complexities inherent in satellite imagery. As a result, existing approaches often do not perform well when applied to satellite data. AI/ML experts whose research uses earth observation data for understanding agricultural issues such as food security have noted that satellite imagery should be considered a “distinct modality for machine learning research and that we must recognize it as such to advance the quality and impact of SatML research across theory, methods, and deployment” (1).
Acknowledging that satellite imagery has distinct characteristics that must be considered in the development of ML models leads to a number of areas for deeper work. A key area is in data used to train ML models. “Most techniques are not generalizable across heterogeneous landscapes” (2)(eg the Amazon Basin and the US Corn Belt have distinctly different signatures in imagery). There is currently a lack of large, high-quality, labeled training data available on a global scale. The investment in making geospatial training datasets that are easy to work with for ML researchers is dwarfed by the investment in mainstream ML research.
In general ML research, the research and development of ImageNet has been credited as a fundamental catalyst in the current boom in AI work. According to their website, ImageNet is “an ongoing research effort to provide researchers around the world with image data for training large-scale object recognition models.” Fei Fei Li, the computer science researcher who founded work on ImageNet, recognized that “the best algorithm wouldn’t work well if the data it learned from didn’t reflect the real world. Her solution: build a better dataset” (3). “One thing ImageNet changed in the field of AI is suddenly people realized the thankless work of making a dataset was at the core of AI research,” said Li. The creation and use of ImageNet has had a profound impact on the technology landscape and is often cited as the primary driver that enabled the current commercial boom in AI.
Essentially, ImageNet provided an expansive training dataset that allowed for benchmarks and model performance improvements. This led to rapid advancements, accelerated research, and commercial applications. Potentially even more impactful was the community that emerged around ImageNet that encourages open science, sharing best practices in model development and benchmarks, and collaborating on new architectures and optimization techniques. ImageNet was followed by more benchmark datasets that laid the foundation for accelerated research in other areas of machine learning and computer vision, such as the Common Objects in Context (COCO) dataset for object detection and segmentation or the Cityscapes dataset for segmentation of urban street scenes.
A number of R&D efforts have chipped away at this problem in the geospatial realm by creating labeled training datasets for geospatial AI research including SpaceNet, RapidAI4EO, AI4Boundaries, Worldstrat, and new efforts like Satellogic EarthView.
These initiatives released valuable training datasets, yet there are still challenges that need focused attention to create the landscape for a commercial boom in geospatial AI:
- Training data efforts are local or regional scale, usually in Europe and the US: Most datasets are not representative of every region of the world. This limits the performance of models in regions that are data sparse.
- Data is from many different sources, lacking interoperability: Geospatial datasets are in many different formats (e.g., different sensors, processing levels, and dimensions) and there are no standards for labeling or sharing training data. This requires custom scripting and large efforts to make use of the data.
- Training datasets are released without community feedback loops or tools for other contributions: Mechanisms for organizations with valuable training data to add to an open dataset would improve geographic coverage, reduce efforts to improve model performance, and enable practitioners to collaborate.
TGE intends to seed progress on these challenges and contribute to the foundation needed for a boom in geospatial AI. Our hope is that more researchers and commercial teams can focus on novel problems with remote sensing data, instead of spending vast amounts of time to get the data ‘ready’. In a talk at Washington University St. Louis titled “Unlocking the Potential of Planetary-Scale Machine Learning for a Sustainable Future,” Dr. Hannah Kerner posed a question thought by many ML researchers using satellite imagery: “where is my ImageNet pre-trained ResNet for remote sensing?” Inspired by this question and the many nods from the audience of researchers desiring such pre-trained models for satellite imagery, TGE has engaged Hannah’s research team along with Dr Nathan Jacobs in our first Innovation Bridge initiative. Both are AI/ML experts who collaborate on efforts to develop AI for Satellite Imagery. Our next post will dive into more specifics of these joint efforts.
- Mission Critical — Satellite Data is a Distinct Modality in Machine Learning; Esther Rolf, Konstantin Klemmer, Caleb Robinson, Hannah Kerner, Feb 2024; https://arxiv.org/pdf/2402.01444
- Supporting Food Security in Africa Using Machine Learning and Earth Observations; Catherine Nakalembe and Hannah Kerner, NASA Harvest, 2022; https://youtu.be/QS0YThiTSsM?si=rQt14FCYw0YmWFV
- https://qz.com/1034972/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world
- Dhakal, A., Ahmad, A., Khanal, S., Sastry, S., Kerner, H., and Jacobs, N. (2024). Sat2Cap: Mapping Fine-Grained Textual Descriptions from Satellite Images. To appear in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, EarthVision 2024
