ChatGeoPT: Exploring the future of talking to our maps

In this blog post we walk through some of our initial explorations using AI interfaces to solve geospatial problems. As a vehicle for trying out these ideas, we’ve created ChatGeoPT, a toy Geospatial AI Assistant. ChatGeoPT translates natural language text into OpenStreetMap Overpass API calls, reads the API response data, and translates the data into natural languge and actionable insights. It also remembers the history of a conversation, allowing for iteration and collaborative problem solving. Use of ChatGeoPT reveals the promise and potential pitfalls of AI-powered geospatial applications.

Background

Since ChatGPT was introduced in late 2022, the AI-powered chatbot has exploded in popularity, becoming the fastest-ever consumer technology to reach 100 million users. There are many reasons behind its rapid adoption, but one way to summarize might be that ChatGPT makes the internet’s data “accessible and actionable.”

Here at the Earth Genome, we seek to do something similar — but with a specific focus on the environment. As you can see on our website, our organizing mission is to make “environmental data accessible and actionable.” So it probably shouldn’t come as any surprise that shortly after starting to use ChatGPT, we began asking ourselves a question: “What would ChatGPT look like, if it were designed for the environment?”

In this blog post, we’ll explore this line of inquiry as applied to the specific problem of geospatial search, a key part of understanding the built and natural environment. In the process we will explore questions like:

• How can natural language be translated into the language of geospatial data science (e.g. API queries)?
• How does ongoing “chat-style” interaction enhance the experience of geospatial search?
• Can we leverage the wide knowledgebase and capabilities of LLMs like GPT-3 to push the limits, by bringing in other factual knowledge and AI-powered creativity?

While there is certainly a long way to go before we get to a true “ChatGPT for the environment,” we’re excited to share what we have learned so far.

Initial Exploration

Like many technical users of ChatGPT, we’ve been impressed by the capabilities of the platform to assist in our everyday development needs. We had heard that GPT-3 was a useful tool for translating text to SQL queries, so we started digging into its abilities in the geospatial realm. Somewhat to our surprise, ChatGPT has a functional knowledge of many geospatial tools, languages, and libraries including:

• PostGIS (“Generate a PostGIS query using Table Highways and Table Admin_Boundaries to determine the length of roads in Washington County, MN”)
• Google Earth Engine (“Write an Earth Engine script to detect NDVI changes over Venezuela”)
• QGIS/PyQGIS (“Write a QGIS script to perform a spatial intersection of two layers and save the result to disk”)
• Geopandas (“Project the dataframe to UTM10N and determine the summed area of all polygons”)
• Overpass QL and XML (“Generate an Overpass QL query for hawker centers in Singapore”)

…and the list goes on and on. In fact, we have yet to find a single widely used geospatial technology that GPT-3 doesn’t know something about. While the code produced by ChatGPT sometimes has flaws, it nevertheless is usually a great place to start.

A deep dive into OpenStreetMap and the Overpass API

We were particularly excited when we realized that ChatGPT could offer assistance in navigating the Overpass API, the programmatic entry point for querying OpenStreetMap’s database. We are in good company in the discovery that ChatGPT understands how to query Overpass better than we do! As an open geospatial database maintained by a wide network of active contributors, OpenStreetMap offers a vast trove of location-specific geographic information, but querying OSM’s database through Overpass is difficult. Not only do you need to be familiar with the somewhat difficult-to-decipher syntax and logic of Overpass QL or XML (the Overpass API languages), but you need to be familiar with the wide variety of tags that are used to label objects in the OSM database.

Thankfully, ChatGPT offers assistance. It not only has knowledge of the relevant tags for a particular search, but it also can compile these into a syntactically correct Overpass QL query.

We even noticed that ChatGPT could output valid links to run the query using Overpass Turbo!

It should be noted that ChatGPT has some significant limitations when it comes to generating Overpass queries. For more sophisticated questions requiring additional logic, it struggles to get the syntax right.

The story gets more interesting, however. Through chat-style interaction, ChatGPT offers a chance to recognize and improve on these mistakes. In other words, if the query suggested by ChatGPT contains a mistake, it’s possible to feed that information back to ChatGPT and help it improve. We believe that this collaborative, “conversational” interaction represents a significant new paradigm in how to approach geospatial problem solving.

The icing on the geospatial cake? ChatGPT can also make sense of the API response. Hand it a JSON string generated by the Overpass API, and it can help you decipher its contents.

Note in the above response, ChatGPT inserts a little extra information that it gleaned from referencing the API response against its own built-in knowledge. The API response for “Round Robin Bar” includes a website address, but does not include the street address or the fact that it is located within the Willard InterContinental. GPT-3 has previously learned this information from studying its training data, and supplies it here when it is relevant.

It should be noted that this injection of built-in knowledge can be a double-edged sword. As many other users of LLMs have noted, LLMs will occasionally “hallucinate” information and present it as fact. Overcoming this tendency is an essential part of deploying any technology built on top of LLMs. We don’t have the magic bullet for this problem (in fact, no one does!) but in this case, we believe that the risk can be reduced by insisting that the LLM rely primarily on the textual information provided in the API response, which has been vetted by the community of OSM contributors.

Building a chat-style Overpass API Assistant

Building on our initial success at using ChatGPT for generating Overpass queries, we decided to take things a few steps further, and build a custom, chat-style Overpass API Assistant. Thankfully, we didn’t need to train our own custom Large Language Model; instead, using OpenAI’s tools, we can simply provide instructions to GPT-3 and have it function as our expert assistant.

The instructions we provided to GPT-3 (specifically, the davinci-003 model) were as follows:

Assistant is an expert OpenStreetMap Overpass API assistant.
For each question that the user supplies, the assistant will reply with:
(1) A statement consenting to help.
(2) The text of a valid Overpass API query that can be used to answer the question. The query should be enclosed by three backticks on new lines, denoting that it is a code block.
(3) A fun fact relating to the question, or a very funny joke or pun related to the question. The joke or pun should also relate to maps, geospatial tech, geography or similar concepts. There is no need to label the fact, joke, or pun.
Assistant has a whimsical personality. Assistant will reply with a geospatial themed joke or a pun if the user asks a question that is not relevant to the Overpass API.

We then took these instructions and packaged them into a chat-style ongoing conversation with GPT-3. Instructing GPT-3 to return the Overpass API query in a code block allows for easy programmatic extraction of the query, which is then executed using the Overpass API. We then feed the API response back into another call to GPT-3 with the request that the model provide a concise answer to the user’s original question.

The result? ChatGeoPT, the world’s first AI assistant for geospatial search.

A few examples of ChatGeoPT

In the examples below, we showcase some examples where ChatGeoPT is able to help solve problems stated in natural language.

Searching for a place to play soccer in the evenings in Baltimore.

Looking for recommendations in Cape Town

Finding a place to celebrate Grandma (note this example shows the conversational chat capability across two screenshots, where we refine the search)

Next steps

ChatGeoPT, in its current form, has many limitations. A few areas where we are excited to push development forward include

• More sophisticated Overpass QL queries. ChatGeoPT is limited to generating very basic Overpass queries. More sophisticated queries, like “Find me all the communities in Schleswig-Holstein without a fire station,” are difficult for ChatGeoPT to generate. Switching to recently released GPT-4 models might help in this regard, as evidenced by some recent discoveries made public by Twitter user John Wiseman (@lemondoor). But even before upgrading, we could provide better feedback to the model (in chat) or fine-tune a GPT-3 model on Overpass API examples.
• Multimodal interactions. There is currently only one way to interact with our AI assistant: the chatbox. In the geographic context, more natural and interesting interactions might be enabled if we allow the user to interact with the assistant through map manipulations.
• Incorporation of other data sources. We see ChatGeoPT as just a first step. Ultimately, we hope to build something like a Large Earth Observation Model (LEOM) that can intelligently interact with and summarize geospatial data sources beyond OSM data — things like satellite imagery, weather data, and anything else that can be plotted on a map.

Key findings

There is so much we have yet to discover about the potential role of LLMs in geospatial workflows. At the outset, we can summarize our major findings as follows:

1. LLMs can be an effective bridge between human language and the language of technical geospatial workflows. Information flows across the bridge both ways: ChatGeoPT is capable of translating natural language into API queries and translating API responses back into easy-to-understand, actionable summaries.
2. Conversational chat can enhance the quality of a geospatial search. We were a little surprised that conversational interactions with an AI assistant could be so valuable a part of the ChatGeoPT experience. Not only is it more enjoyable to have a conversation with an assistant that has a “personality,” but chat also enables feedback — either to refine a search or (perhaps more interestingly) to correct a mistake made by the model.
3. Injection of built-in LLM knowledge is both a powerful opportunity and a potential risk. LLMs like GPT-3 have an immense amount of built-in knowledge. Not only does this allow them to generate things like Overpass queries without any specific training, it also allows them to spontaneously generate things like a “bounding box for Boston.” However, the potential of LLMs to hallucinate is a clear risk for using outputs of any tech that relies on their outputs.

The future of talking to our maps

With ChatGeoPT, we have just scratched the surface of what is possible by applying the latest generation of LLMs to geospatial problems. We’re excited to see where this goes! If you’re interested in joining us on the journey, please check out our GitHub, tweet us your thoughts, and comment on this post.