Stateful routing with LangGraph

Routing like a call center

When calling a call center you may often be redirected from one agent to another. Some may be human. Some may be robots.

Chatbot systems often need to follow a similar pattern. Just like with the human, it is impossible to tell one agent how to do all the things. A chatbot system with multiple agents needs to route requests from one agent to another.

I call it stateful routing because every new request must be redirected to the last agent you’ve spoken to. In other words, the route is memorized in the graph state.

Consider this dialog

Human: Hello

Greeting_Agent: Hello. I'm an automated assistant. How can I help you today?

Human: I'd like to open an account

Sign_up_Agent: Great! I'll need to collect some information to get started. Could you please provide me with your first name, last name, and email address?

Human: Sure.  My name is John Doe.

Sign_up_agent: Thank you, John. Could you please provide your email address?

Human: john.doe@example.com

Sign_up_Agent: Thank you, John! Your account has been created, and a verification email has been sent to john.doe@example.com. Please check your email to complete the verification process.
If you need any further assistance, feel free to ask!

Human: Thank you

Greeting_Agent: You're welcome! If you have any other questions or need further assistance in the future, don't hesitate to reach out. Have a great day!

Notice that once we identify the user's intent they are directed to a specialized Sign_up_Agent agent and the conversation continues with that agent until the sign-up process is complete.

Tech Stack

For the system described above I will be using LangGraph. Which gives more low-level control over the agentic flow than plain LangChain.

I will use LangGraph Studio for visualization and testing.

In this article, I will not dive into how LangGraph works. There are great resources out there

• LangChain academy
• DeepLearning course

For inpatients here is a link to the git repo

Requirements

Let’s design a chatbot system that works like this.

• A new user is met by the Greeting_Agent. It must ask the user about their inquiry and redirect to an appropriate agent.
• Report_Agent Ask the user about an issue and submit the report.
• Sign_up_Agent Signs up as a new user asking it some personal data.
• Once the user is redirected to an agent they continue to talk bypassing the Greeting_Agent.
• Once an agent is done with their task they redirect back to the Greeting_Agent

Implementation

Pre greeting routing

By default, the new message is routed to the first node in the graph. But it is possible to squeeze a conditional edge after the START node.

This routing edge routes to the agent defined in current_route property in the graph state or to the Greeting_Agent by default.

# Define graph state
class AgentState(MessagesState):
    current_route: str

# Routs to the last used agent.
def pre_greeting_routing (state:AgentState) -> str :
        # Route to the last used agent, if it exists in the state
        if 'current_route' in state and state['current_route']:
            return state['current_route']
        else:
            # Default to the Greeting Agent if no current_route is found
            return GREETING_AGENT
    
    return routing

# Build graph
builder = StateGraph(AgentState)
builder.add_conditional_edges(
    START,
    pre_greeting_routing,
)

Greeting Agent

Now we needGreeting_Agent to decide where to redirect the request. This can be simply done by providing it with a redirect_tool
and some prompt engineering.

greeting_agent_prompt = f"""
    You are an agent in an online store. 
    You are part of a team of other agent that can perform more specialized tasks.
    You are the first in the chain of agents. 
    You goal is to greet the customer and identify their needs.
    Once you understand the user question use redirect_tool to redirect the user the specialized agent.
    
    There are following agent that you can redirect to:
    - {REPORT_AGENT} : Can submit an issue to the tech support
    - {SIGN_UP_AGENT} : Can sing up a new user
        
    examples:
        user: Hello
        agent: Hello. I'm automated assistant. How can I help you? 
        user: I'd like to open an account
        tool_call: redirect_tool
    """

The tricky part here is that Tool nodes in LangGraph can not mutate the state. And thus a tool itself can not redirect to another agent. Yes, it is possible to inject the state into the tool but updates will not be visible to the rest of the graph.

But tools in LangGraph can not only return a text response but also an artifact. We will use the artifact to propagate the state update from the tool to an agent that will apply it.

# defining a tool with `content_and_artifact` return
@tool(parse_docstring=True, response_format="content_and_artifact")
def redirect_tool(
    next_agent: str, 
) -> dict:
    """A tool that redirects to a specific agent.
    
    Args:
        next_agent: Name of the agent to redirect to.
    """
    # Returning a text response and updated value for the state
    return f"You will be redirected to {next_agent}", {'current_route': next_agent}

This artifact will then be processed by the Greeting_Agentwhich will apply to the state.

• Make a request to LLM
• If LLM responds with a tool call — call it.
• The tool returns a ToolMessage and a state update. Apply it.

This is almost like a ReAct agent created by create_react_agent with a difference that it also needs to update the graph state.

from typing import Callable, List
from langgraph.prebuilt import tools_condition, ToolNode
from langchain_core.messages import SystemMessage
from langgraph.prebuilt import ToolNode

def create_tool_calling_agent(llm, system_promt:str, agent_name: str, tools: List[Callable], call_after_tool: bool = True):
    llm_with_tools = llm.bind_tools(tools)

    def agent (state, config):
        # invoke agent
        llm_response = llm_with_tools.invoke([SystemMessage(system_promt)] + state["messages"])
        llm_response.name = agent_name
        # add AIMessage to the state
        state['messages'].append(llm_response)

        # use tools_condition to detect if a tool is called
        if(tools_condition(state) == 'tools'):
            # creating a tool node on the fly but not adding it to the graph
            tool_node = ToolNode(tools)
            response = tool_node.invoke(state)

            # adding all the ToolMessage
            for tool_message in response['messages']:
                state['messages'].append(tool_message)
                # applying state update received from tool
                if tool_message.artifact:
                    state={**state, **tool_message.artifact}

            # there are cases when we may or may not want to call the same agent after tool
            if call_after_tool :
                return agent(state, config)
            
        return state

greeting_agent = create_tool_calling_agent(llm, greeting_agent_prompt, GREETING_AGENT, [redirect_tool], call_after_tool=False)

Once the Greeting_Agent decides to pass control to another agent it calls redirect_tool with the next_agent parameter. This tool does nothing. It only returns a state update {'current_route':next_agent}.
Greeting_Agent applies the state with the new route in it.

Tool nodes are not added directly to the graph but are contained within an agent, which makes the graph much simpler.

Let's consider this dialog.

User: Hello
Greeting_Agent: Hello how can I help you
User: I’d like to open an account

The phrase I’d like to open an account triggers redirect_tool call. And we’d like this message to be passed to the Sign_up_Agent and not back to the Greetint_Agent.Notice the parameter call_after_tool in the snippet above. It regulates whether the tool response is passed back to the agent or not.
We will use it for our other agents too.

Post greeting redirect

After graph leaves Greeting_Agent we put another conditional edge but with a slightly different logic.

# Routes to a new selected agent.
def post_greeting_routing (state:AgentState) -> str :
    # if current agent is greeting, route to the END
    if 'current_route' not in state or state['current_route'] == GREETING_AGENT:
        return END
    elif 'current_route' in state and state['current_route']:
        return state['current_route']

builder.add_conditional_edges(
    GREETING_AGENT,
    post_greeting_routing,
)

Agents

Other agents Report_Agent and Sign_up_Agent follow almost the same pattern as Greeting_Agnet. They chat with the human until they collect all the data to fulfill their purpose which is to call a tool.

from langchain_core.tools import tool

def save_report_tool(return_route:str):
    @tool(parse_docstring=True, response_format="content_and_artifact")
    def tool_func(report: str) -> str:
        """Saves an issue reported by the user.
        
        Args:
            issue: Summary of the issue reported by the user
        """
        print("SUBMITING REPORT" , report)
        
        return "report is submitted", {'current_route': return_route, 'reports': [report]}

    return tool_func


report_agent_prompt = f"""You are the assistant who receives a report about an issue. 
Ask the user to describe an issue. Once done use the tools available to save it.
Example:
User: My internet is very slow
Assistant: I\'m sorry to hear that. Can you please provide me with more details about the issue?
User: I can not tell you much. Youtube is just not working.
Assistant: Thank you for the information. Have notified the support team.
"""

def sign_up_tool(return_route:str):
    
    @tool(parse_docstring=True, response_format="content_and_artifact")
    def tool_func(first_name: str, last_name: str, email: str):
        """Saves user first and last name.
        
        Args:
            first_name : user first name
            last_name : user last name
            email : user email
        """
        print("CREATING USER" , first_name, last_name, email)
        
        return "User signed up. Verification email is sent", {
            'current_route': return_route, 
            'first_name': first_name, 
            'last_name': last_name, 
            'email': email
            }
    
    return tool_func


sign_up_prompt = f"""
You are the assistant that signs up a user. 
You need to collect user name, last name, and email and save it using tools
"""

sign_up_agent = create_tool_calling_agent(llm, sign_up_prompt, SIGN_UP_AGENT, [sign_up_tool(GREETING_AGENT)])
report_agent = create_tool_calling_agent(llm, report_agent_prompt, REPORT_AGENT, [save_report_tool(GREETING_AGENT)])

Notice that I used the same create_tool_calling_agent function as with Greeting_Agent the only difference that the call_after_tool parameter is defaulted to True. It controls whether an agent replies after a tool call or not.

Other Techniques for Humans in the Loop.

There are several other approaches for humans in the loop that LangGraph offers. interupt_before, interupet_after, dynamic breakpoints with NodeInterupts, or utilize conditional edge to continue or interrupt.
Thou it is possible to achieve the same routing behavior with these techniques I find it less convenient to work with. Comparing with each of them will bloat the article. And there will be applications when they shine.

Code

Please find the full code here.

You can test the graph in a Jupiter notebook or with a simple Python script. Or load it into a LangGrpah studio. All necessary files are inside.

Feedback

Since Agentic systems is a relatively new field the patterns are still developing. Please use the comment to leave your thoughts. What approach would you use?