Midstream research note: Rebuilding the Earth Index user interface

Written by Zoe Statman-Weil, Mikey Abela and Mikel Maron

We are rebuilding the Earth Index user interface so anyone can quickly find evidence of environmental issues in any region without deploying and training complicated compute-heavy models or laboriously examining remote sensing imagery manually.

Sharing this midstream research note is a key part of transparency of our process. The use of Earth Index is developed in close collaboration with users. Distributing our work and learnings as we go will help Earth Index be better, and also we hope raise the game of all efforts in this space.

Every step of Earth Index development has been scrappy and iterative. The UI of Earth Index reflected that, and it has been in need of a rethink. We’re thankful to the Rockefeller Foundation and Patrick J. McGovern Foundation for supporting development of a new interface to aid the work of indigenous groups and all Earth Index uses.

Starting with a Notebook

The Earth Index UI started out as a Jupyter notebook leveraging iPyleaflet to create an interactive experience and Google Earth Engine for data generation and image visualization. These tools got us shockingly far, allowing our internal team to conduct searches to find similar earth surface features as well as identify patterns of change. We iterated quickly, experimenting with tools that could help us answer our partners questions quickly and without the need for training a large ML model. Through the use of iPyleaflet we were able to build a UI that could:

• Select a collection of embeddings, a region of interest, and a time period of interest
• Select positive and negative points with various basemap options such as Mapbox, Google or Sentinel-2
• Display nearest neighbor search results on a map
• Cull through those results one by one or label a group at a time
• Visualize a time-series of images at specific location
• Graph nearest neighbor search results in embedding space
• Narrow down results with a light-weight classifier model

While the notebook helped our internal team solve problems locally, the version we hosted with voila was slow and hard to debug and thus not easy to share with partners and users. As there was no API, the “frontend” and “backend” code ended up in a large codebase that could feel hard to navigate and hard to adapt to changing requirements. A user-friendly app required a shift in development approach, an API, and a real frontend.

Development of the API

We chose to write the primary search API in Go because of Go’s performance, simplicity and the team’s experience building data-heavy APIs like Climate TRACE in Go. The goal of the initial API was to allow for simple similarity search and labeling. It would serve as a layer between the frontend and the vector database. As discussed in a previous post, the vector search engine / database world is rapidly developing so while we are using Qdrant now, the code needs to be flexible enough to substitute Qdrant for another vector database. Capturing the desired database methods with a Go interface is a good way to abstract away the underlying database logic. Now no matter the database we use, if we can perform these methods, none of the handlers or API responses will need to change.

// VectorDatabase defines the interface for any vector database service

type VectorDatabase interface {
  Connect() error
  Close() error
  Cancel()
  GetMatchPointsFromSearchPoints(collection models.Collection,
     timePeriods []models.TimePeriod,
     searchPoints []models.SearchPointFeature,
     wEmbedding bool) ([]models.MatchPoint, error)
  Search(collection models.Collection,
     geoID int,
     timePeriod []models.TimePeriod,
     queryVector *models.Embedding,
     wEmbeddings bool,
     searchNum int) ([]models.MatchPoint, error)
  GetEmbeddingsFromPointIds(
     collection models.Collection,
     points []string) ([]models.Embedding, error)

While Go was chosen to build the heart of the backend for the UI, Python is too crucial to data science and geospatial to leave behind. We are using Python services to serve the basemap imagery and run a model (e.g., random forest classifier) to cull down results from a nearest neighbor search.

With a real API, we can now meet new user requirements in the code in a clean and adaptable manner.

An API based proof of concept

The frontend was built using the Next.js React framework with Typescript. React has an incredibly robust development community around it. We could quickly incorporate UI component libraries like Mantine UI, map integrations like Mapbox GL JS, and authentication providers like Clerk to speed the development process. Next.js introduces additional functionality to React with features like built-in routing, client and server components, and middleware which make for a more organized, performant, and secure web application.

As we developed the frontend, we were able to adapt the search API design as needed. Given that we were providing GeoJSON sources to Mapbox in order to create interactive map layers, we designed the API responses as GeoJSON feature collections. When we realized that Javascript couldn’t support uint64 ids on our points, we made the point ids into strings.

Ultimately, we were able to arrive at a full-stack web application that supports simple feature detection search. The new application covers only a subset of the previous notebook UI’s features as of now, but it is much more responsive and allows for independent development and deployment of the backend and frontend.

With the search API now incorporated into a simple UI, we were eager to turn to real users to determine in which direction we should take Earth Index.

What we’ve learned from talking to users and where we’re going

We spoke with data specialists, technical leaders, and developers who work closely with indigenous communities to understand their use cases. There were several commonly requested features that we gleaned from these conversations:

• Automated monitoring and alerting on specific regions of interest
• Generation of data analytics from searches
• Authentication and session management

For a Yanomami indigenous community member in the Brazilian Amazon, whose land is often threatened by illegal gold mining, we can see why these features would be useful. They want to quickly be notified of any new illegal activity in their community so that they may take action and notify Brazilian authorities. To ensure that authorities will take action, they want images and figures that prove the illegal activity is occurring. And they want to be able to return to the same search on their profile as new data becomes available for the same region. These are features which we are excited to build into Earth Index.

We also want to optimize the existing search experience for users. With the functionality in place, we are returning to the wireframes to design a more intentional and guided point-labeling process. As we do so, we are putting these designs in front of real users to get feedback and to ground the UX in their use cases.

If you are interested in being an early user, providing feedback, or hearing more about Earth Index, you can sign up for the waitlist and / or reach out to us.