Building a Smart Travel Itinerary Suggester with LangChain, Google Maps API, and Gradio (Part 1)
Learn how to build an application that might inspire your next road trip
This article is part 1 of a three part series where we build a travel itinerary suggester application using OpenAI and Google APIs and display it in a simple UI generated with gradio. In this part, we start by discussing prompt engineering for this project. Just want see the code? Find it here.
1. Motivation
Since the launch of ChatGPT in late 2022, there has been an explosion of interest in large language models (LLMs) and their application in consumer-facing products such as chatbots and search engines. Less than a year later, we have access to a plethora of open source LLMs available from model hubs such as Hugging Face, model hosting services such as Lamini and paid APIs such as OpenAI and PaLM. It is both exciting and somewhat overwhelming to see how fast this field is advancing, with new tools and development paradigms seemingly emerging every few weeks.
Here we’ll be sampling just a fraction of this zoo of tools to build a useful application that could help us with travel planning. When planning a vacation its often nice to get suggestions from someone who’s been there before, and even better to see those suggestions laid out on a map. In the absence of this advice, sometimes I’ll just browse Google Maps in the general area that I want to visit and haphazardly select a few places that look interesting. Maybe this process is fun, but its inefficient and likely to miss something. Wouldn’t it be nice to have a tool that could give you a bunch of suggestions with just a few high level preferences?
Thats exactly what we’ll try to build: A system that can provide travel itinerary suggestions given some high level preferences, something like “I have 3 days to explore the San Francisco and love art museums”. Google Search’s generative AI feature and ChatGPT can already produce creative results for queries like this, but we want to go a step further and produce an actual itinerary with travel times and a nice map to help the user get orientated.
The goal is more to get acquainted with the tools needed to build a service like this rather than actually deploy the application, but along the way we’ll learn a bit about prompt engineering, LLM orchestration with LangChain, using the Google Maps API to extract directions and displaying the results with leafmap and gradio. It’s amazing how quickly these tools allow you to build a POC for systems like this, but as always the real challenges lie in evaluation and edge case management. The tool we’ll build is far from perfect and if anyone is interested in helping me develop it further that would be fantastic.
2. Prompting strategy
This project will make use of the OpenAI and Google PaLM API. You can get make API keys by making accounts here and here. At the time of writing, the Google API has limited general availability and has a waitlist, but it should only take a few days to get access.
Use of dotenv is an easy way to avoid having to copy and paste API keys into your development environment. After making a .env file with the following lines
OPENAI_API_KEY = {your open ai key}
GOOGLE_PALM_API_KEY = {your google palm api key}We can use this function to load the variables ready for downstream use by LangChain for example
from dotenv import load_dotenv
from pathlib import Path
def load_secets():
load_dotenv()
env_path = Path(".") / ".env"
load_dotenv(dotenv_path=env_path)
open_ai_key = os.getenv("OPENAI_API_KEY")
google_palm_key = os.getenv("GOOGLE_PALM_API_KEY")
return {
"OPENAI_API_KEY": open_ai_key,
"GOOGLE_PALM_API_KEY": google_palm_key,
}Now, how should we design the prompts for a travel agent service? The user will be free to enter any text they want, so we first want to be able to determine whether or not their query is valid. We definitely want to flag any query that contain harmful content, such requests for an itinerary with malicious intent.
We also want to filter out questions unrelated to travel — no doubt the LLM could provide an answer to such questions, but they are beyond the scope of this project. Finally, we also want to identify requests that are unreasonable like “I want to fly to the moon” or “I want to do a three day road trip from New York to Tokyo”. Given an unreasonable request like this, it would be great of the model could explain why it was unreasonable and suggest a modification that would help.
Once the request is validated we can proceed to providing a suggested itinerary, which ideally should contain specific addresses of the waypoints so that they can be sent to a mapping or directions API such as Google Maps.
The itinerary should be human-readable, with enough detail for the user to to find it useful as a stand-alone suggestion. Large, instruction-tuned LLMs such as ChatGPT seem to be great at providing such responses, but we need to make sure that the waypoint addresses are extracted in a consistent way.
So there are three distinct stages here:
- Validate the query
- Produce the itinerary
- Extract the waypoints in a format that can be understood by the Google Maps API
It may be possible to design a prompt that can do all three in one call, but for ease of debugging we will split them up into three LLM calls, one for each part.
Fortunately LangChain’s PydanticOutputParser can really help here, by providing a set of pre-made prompts that encourage LLMs to format their responses in a way that conforms with an output schema.
3. The validation prompt
Let’s take a look at the validation prompt, which we can wrap in a template class to make it easier to contain and iterate on different versions.
from langchain.prompts.chat import (
ChatPromptTemplate,
SystemMessagePromptTemplate,
HumanMessagePromptTemplate,
)
from langchain.output_parsers import PydanticOutputParser
from pydantic import BaseModel, Field
class Validation(BaseModel):
plan_is_valid: str = Field(
description="This field is 'yes' if the plan is feasible, 'no' otherwise"
)
updated_request: str = Field(description="Your update to the plan")
class ValidationTemplate(object):
def __init__(self):
self.system_template = """
You are a travel agent who helps users make exciting travel plans.
The user's request will be denoted by four hashtags. Determine if the user's
request is reasonable and achievable within the constraints they set.
A valid request should contain the following:
- A start and end location
- A trip duration that is reasonable given the start and end location
- Some other details, like the user's interests and/or preferred mode of transport
Any request that contains potentially harmful activities is not valid, regardless of what
other details are provided.
If the request is not valid, set
plan_is_valid = 0 and use your travel expertise to update the request to make it valid,
keeping your revised request shorter than 100 words.
If the request seems reasonable, then set plan_is_valid = 1 and
don't revise the request.
{format_instructions}
"""
self.human_template = """
####{query}####
"""
self.parser = PydanticOutputParser(pydantic_object=Validation)
self.system_message_prompt = SystemMessagePromptTemplate.from_template(
self.system_template,
partial_variables={
"format_instructions": self.parser.get_format_instructions()
},
)
self.human_message_prompt = HumanMessagePromptTemplate.from_template(
self.human_template, input_variables=["query"]
)
self.chat_prompt = ChatPromptTemplate.from_messages(
[self.system_message_prompt, self.human_message_prompt]
)Our Validation class contains the output schema definitions for the query, which will be a JSON object with two keys plan_is_valid and updated_request. InsideValidationTemplate we use LangChain’s helpful template classes to construct our prompt and also create a parser object with PydanicOutputParser. This converts the Pydantic code in Validation into a set of instructions that can be passed to the LLM along with the query. We can then include reference to these format instructions in the system template. Every time the API is called, we want the both the system_message_promptand the human_message_prompt to be sent to the LLM, which is why we package them together in the chat_prompt.
Since this isn’t really a chatbot application (although it could be made into one!) we could just put both the system and human templates into the same string and get the same response.
Now, we can make an Agent class that uses LangChain to call the LLM API with the template defined above. Here we’re using ChatOpenAI , but it can be replaced with GooglePalm if you prefer that.
Note that we also make use of LLMChain and SequentialChain from Langchain here, even though we’re only making a single call to the LLM. This is probably overkill, but it could be helpful for extensibility in future of we wanted to add another call , for example to the OpenAI moderation API before the validation chain runs.
import openai
import logging
import time
# for Palm
from langchain.llms import GooglePalm
# for OpenAI
from langchain.chat_models import ChatOpenAI
from langchain.chains import LLMChain, SequentialChain
logging.basicConfig(level=logging.INFO)
class Agent(object):
def __init__(
self,
open_ai_api_key,
model="gpt-3.5-turbo",
temperature=0,
debug=True,
):
self.logger = logging.getLogger(__name__)
self.logger.setLevel(logging.INFO)
self._openai_key = open_ai_api_key
self.chat_model = ChatOpenAI(model=model, temperature=temperature, openai_api_key=self._openai_key)
self.validation_prompt = ValidationTemplate()
self.validation_chain = self._set_up_validation_chain(debug)
def _set_up_validation_chain(self, debug=True):
# make validation agent chain
validation_agent = LLMChain(
llm=self.chat_model,
prompt=self.validation_prompt.chat_prompt,
output_parser=self.validation_prompt.parser,
output_key="validation_output",
verbose=debug,
)
# add to sequential chain
overall_chain = SequentialChain(
chains=[validation_agent],
input_variables=["query", "format_instructions"],
output_variables=["validation_output"],
verbose=debug,
)
return overall_chain
def validate_travel(self, query):
self.logger.info("Validating query")
t1 = time.time()
self.logger.info(
"Calling validation (model is {}) on user input".format(
self.chat_model.model_name
)
)
validation_result = self.validation_chain(
{
"query": query,
"format_instructions": self.validation_prompt.parser.get_format_instructions(),
}
)
validation_test = validation_result["validation_output"].dict()
t2 = time.time()
self.logger.info("Time to validate request: {}".format(round(t2 - t1, 2)))
return validation_testTo run an example, we can try the following code. Setting debug=True will activate LangChain’s debug mode, which prints the progression of the query text at is moves though the various LangChain classes on its way too and from the LLM call.
secrets = load_secets()
travel_agent = Agent(open_ai_api_key=secrets[OPENAI_API_KEY],debug=True)
query = """
I want to do a 5 day roadtrip from Cape Town to Pretoria in South Africa.
I want to visit remote locations with mountain views
"""
travel_agent.validate_travel(query)This query seems reasonable, so we get a result like this
INFO:__main__:Validating query
INFO:__main__:Calling validation (model is gpt-3.5-turbo) on user input
INFO:__main__:Time to validate request: 1.08
{'plan_is_valid': 'yes', 'updated_request': ''}Now we test by changing the query to something less reasonable, such as
query = """
I want to walk from Cape Town to Pretoria in South Africa.
I want to visit remote locations with mountain views
"""The response time is longer because ChatGPT is trying to provide an explanation for why the query is not valid and therefore generating more tokens.
INFO:__main__:Validating query
INFO:__main__:Calling validation (model is gpt-3.5-turbo) on user input
INFO:__main__:Time to validate request: 4.12
{'plan_is_valid': 'no',
'updated_request': 'Walking from Cape Town to Pretoria in South Africa is not ...' a4. The itinerary prompt
If a query is valid, it can pass on to the next stage, which is the itinerary prompt. Here, we want to the model to return a detailed suggested travel plan, which should take the form of a bulleted list with waypoint addresses and some advice about what to do in each place. This is really the main “generative” part of the project, and there are many ways to design a query to give good results here. Our ItineraryTemplate looks like this
class ItineraryTemplate(object):
def __init__(self):
self.system_template = """
You are a travel agent who helps users make exciting travel plans.
The user's request will be denoted by four hashtags. Convert the
user's request into a detailed itinerary describing the places
they should visit and the things they should do.
Try to include the specific address of each location.
Remember to take the user's preferences and timeframe into account,
and give them an itinerary that would be fun and doable given their constraints.
Return the itinerary as a bulleted list with clear start and end locations.
Be sure to mention the type of transit for the trip.
If specific start and end locations are not given, choose ones that you think are suitable and give specific addresses.
Your output must be the list and nothing else.
"""
self.human_template = """
####{query}####
"""
self.system_message_prompt = SystemMessagePromptTemplate.from_template(
self.system_template,
)
self.human_message_prompt = HumanMessagePromptTemplate.from_template(
self.human_template, input_variables=["query"]
)
self.chat_prompt = ChatPromptTemplate.from_messages(
[self.system_message_prompt, self.human_message_prompt]
)Note that there is no need for a Pydantic parser here because we want the output to be a string rather than a JSON object.
To use this, we can add a new LLMChain to the Agent class, which looks like this
travel_agent = LLMChain(
llm=self.chat_model,
prompt=self.itinerary_prompt.chat_prompt,
verbose=debug,
output_key="agent_suggestion",
)We did not set themax_tokens argument when instantiating the chat_model here, which allows the model to decide the length of its output. With GPT4 in particular this can make the response time rather long (30s+ in some cases). Interestingly, the response times of PaLM are considerably shorter.
5. The waypoint extraction prompt
Using the itinerary prompt might give us a nice list of waypoints, perhaps something like this
- Day 1:
- Start in Berkeley, CA
- Drive to Redwood National and State Parks, CA (1111 Second St, Crescent City, CA 95531)
- Explore the beautiful redwood forests and enjoy nature
- Drive to Eureka, CA (531 2nd St, Eureka, CA 95501)
- Enjoy the local cuisine and explore the charming city
- Overnight in Eureka, CA
- Day 2:
- Start in Eureka, CA
- Drive to Crater Lake National Park, OR (Crater Lake National Park, OR 97604)
- Marvel at the stunning blue lake and hike the scenic trails
- Drive to Bend, OR (Bend, OR 97701)
- Indulge in the local food scene and explore the vibrant city
- Overnight in Bend, OR
- Day 3:
- Start in Bend, OR
- Drive to Mount Rainier National Park, WA (55210 238th Ave E, Ashford, WA 98304)
- Enjoy the breathtaking views of the mountain and hike the trails
- Drive to Tacoma, WA (Tacoma, WA 98402)
- Sample the delicious food options and explore the city's attractions
- Overnight in Tacoma, WA
- Day 4:
- Start in Tacoma, WA
- Drive to Olympic National Park, WA (3002 Mount Angeles Rd, Port Angeles, WA 98362)
- Explore the diverse ecosystems of the park and take in the natural beauty
- Drive to Seattle, WA (Seattle, WA 98101)
- Experience the vibrant food scene and visit popular attractions
- Overnight in Seattle, WA
- Day 5:
- Start in Seattle, WA
- Explore more of the city's attractions and enjoy the local cuisine
- End the trip in Seattle, WANow we need to extract the addresses of the waypoints so that we can proceed to the next step, which is going to be plotting them on a map and calling the Google Maps directions API to obtain directions between them.
To do this, we will make another LLM call and use PydanicOutputParser again to make sure our output is formatted correctly. To understand the format here, it’s useful to briefly consider what we want to do at the next stage of this project (covered in part 2). We will be making a call to the Google Maps Python API, which looks like this
import googlemaps
gmaps = googlemaps.Client(key=google_maps_api_key)
directions_result = gmaps.directions(
start,
end,
waypoints=waypoints,
mode=transit_type,
units="metric",
optimize_waypoints=True,
traffic_model="best_guess",
departure_time=start_time,
)Where start and end are addresses as strings, and waypoints is a list of addresses to be visited in between.
Our requested schema for the waypoint extraction prompt therefore looks like this
class Trip(BaseModel):
start: str = Field(description="start location of trip")
end: str = Field(description="end location of trip")
waypoints: List[str] = Field(description="list of waypoints")
transit: str = Field(description="mode of transportation")Which will enable us plug the outputs of the LLM call into the directions call.
For this prompt, I found that adding a one-shot example really helped the model conform to the desired output. Fine-tuning of a smaller, open source LLM to extract lists of waypoints using these results from ChatGPT/PaLM might be an interesting spinoff project here.
class MappingTemplate(object):
def __init__(self):
self.system_template = """
You an agent who converts detailed travel plans into a simple list of locations.
The itinerary will be denoted by four hashtags. Convert it into
list of places that they should visit. Try to include the specific address of each location.
Your output should always contain the start and end point of the trip, and may also include a list
of waypoints. It should also include a mode of transit. The number of waypoints cannot exceed 20.
If you can't infer the mode of transit, make a best guess given the trip location.
For example:
####
Itinerary for a 2-day driving trip within London:
- Day 1:
- Start at Buckingham Palace (The Mall, London SW1A 1AA)
- Visit the Tower of London (Tower Hill, London EC3N 4AB)
- Explore the British Museum (Great Russell St, Bloomsbury, London WC1B 3DG)
- Enjoy shopping at Oxford Street (Oxford St, London W1C 1JN)
- End the day at Covent Garden (Covent Garden, London WC2E 8RF)
- Day 2:
- Start at Westminster Abbey (20 Deans Yd, Westminster, London SW1P 3PA)
- Visit the Churchill War Rooms (Clive Steps, King Charles St, London SW1A 2AQ)
- Explore the Natural History Museum (Cromwell Rd, Kensington, London SW7 5BD)
- End the trip at the Tower Bridge (Tower Bridge Rd, London SE1 2UP)
#####
Output:
Start: Buckingham Palace, The Mall, London SW1A 1AA
End: Tower Bridge, Tower Bridge Rd, London SE1 2UP
Waypoints: ["Tower of London, Tower Hill, London EC3N 4AB", "British Museum, Great Russell St, Bloomsbury, London WC1B 3DG", "Oxford St, London W1C 1JN", "Covent Garden, London WC2E 8RF","Westminster, London SW1A 0AA", "St. James's Park, London", "Natural History Museum, Cromwell Rd, Kensington, London SW7 5BD"]
Transit: driving
Transit can be only one of the following options: "driving", "train", "bus" or "flight".
{format_instructions}
"""
self.human_template = """
####{agent_suggestion}####
"""
self.parser = PydanticOutputParser(pydantic_object=Trip)
self.system_message_prompt = SystemMessagePromptTemplate.from_template(
self.system_template,
partial_variables={
"format_instructions": self.parser.get_format_instructions()
},
)
self.human_message_prompt = HumanMessagePromptTemplate.from_template(
self.human_template, input_variables=["agent_suggestion"]
)
self.chat_prompt = ChatPromptTemplate.from_messages(
[self.system_message_prompt, self.human_message_prompt]
)Now, lets add a new method to the Agent class that can call the LLM with ItineraryTemplate and MappingTemplate sequentially using SequentialChain
def _set_up_agent_chain(self, debug=True):
# set up LLMChain to get the itinerary as a string
travel_agent = LLMChain(
llm=self.chat_model,
prompt=self.itinerary_prompt.chat_prompt,
verbose=debug,
output_key="agent_suggestion",
)
# set up LLMChain to extract the waypoints as a JSON object
parser = LLMChain(
llm=self.chat_model,
prompt=self.mapping_prompt.chat_prompt,
output_parser=self.mapping_prompt.parser,
verbose=debug,
output_key="mapping_list",
)
# overall chain allows us to call the travel_agent and parser in
# sequence, with labelled outputs.
overall_chain = SequentialChain(
chains=[travel_agent, parser],
input_variables=["query", "format_instructions"],
output_variables=["agent_suggestion", "mapping_list"],
verbose=debug,
)
return overall_chainTo make these calls, we can use the following code
agent_chain = travel_agent._set_up_agent_chain()
mapping_prompt = MappingTemplate()
agent_result = agent_chain(
{
"query": query,
"format_instructions": mapping_prompt.parser.get_format_instructions(),
}
)
trip_suggestion = agent_result["agent_suggestion"]
waypoints_dict = agent_result["mapping_list"].dict()The addresses in waypoints_dict should be sufficiently formatted for use with Google Maps, but they can also be geocoded to reduce the likelihood of errors when calling the directions API. The waypoints dictionary should look something like this.
{
'start': 'Berkeley, CA',
'end': 'Seattle, WA',
'waypoints': [
'Redwood National and State Parks, 1111 Second St, Crescent City, CA 95531',
'Crater Lake National Park, Crater Lake National Park, OR 97604',
'Mount Rainier National Park, 55210 238th Ave E, Ashford, WA 98304',
'Olympic National Park, 3002 Mount Angeles Rd, Port Angeles, WA 98362'
],
'transit': 'driving'
}6. Putting it all together
We now have the ability to use an LLM to validate a travel query, generate a detailed itinerary and extract the waypoints as a JSON object that can be passed down steam. You’ll see that in the code, almost all of this functionality is handled by the Agent class, which is instantiated inside TravelMapperBase and used as follows
travel_agent = Agent(
open_ai_api_key=openai_api_key,
google_palm_api_key=google_palm_api_key,
debug=verbose,
)
itinerary, list_of_places, validation = travel_agent.suggest_travel(query)Using LangChain makes it very easy to swap out the LLM that is being used. For PALM, we simply need to declare
from langchain.llms import GooglePalm
Agent.chat_model = GooglePalm(
model_name="models/text-bison-001",
temperature=0,
google_api_key=google_palm_api_key,
)And for OpenAI, we can either use ChatOpenAI or OpenAI as described in the sections above.
Now, we’re ready to move onto the next stage: How to we convert the list of places into a set of directions and plot them on a map for the user to examine? This will be covered in part 2 of this three part series.
Thanks for reading! Please feel free to explore the full codebase here https://github.com/rmartinshort/travel_mapper. Any suggestions for improvement or extensions to the functionality would be much appreciated!
