Using LLM with ReAct Agent

Nisarg Mehta

8 min readJun 19, 2024

Introduction

LLMs are incredibly powerful, yet they lack particular abilities such as logic, calculation, and search where computers typically excel, but LLMs struggle.
Given the significant limitations of current LLMs, solutions to these challenges must be sought. One potential solution set comes in the form of “agents”.
In this article, we would be discussing in detail about Reasoning and Action (ReAct) Agent.
We’ll begin with an overview of agents. Following this, we’ll delve into the ReAct Agent, understanding its framework through a practical example. After this we will go through Zero-shot ReAct Agent and Conversational ReAct Agent, showcasing its application in interacting with a small stock database.
Finally, we would be going through advantages and limitations of using ReAct Agent.

Agents Overview

Agents can be seen as tools that enhance the capabilities of LLMs, akin to how humans use calculators for math or conduct Google searches for information. Agents enable an LLM to perform various tasks.
An Agent uses an LLM to decide on actions and their sequence. Actions include utilizing a tool, observing its output, or providing feedback to the user.
Agents typically relies on a language model, a prompt, and an output parser. Different agents employ various prompting styles for reasoning, input encoding methods, and output parsing techniques.
The logic of Agents is to follow step-by-step thought process where the advantage is LLM can work through multiple reasoning steps to produce a better answer.

Schema of Agents:

AgentAction: This is a dataclass that represents the action an agent should take.
AgentFinish: This represents the final result from an agent, when it is ready to return to the user.
Intermediate Steps: These represent previous agent actions and corresponding outputs from the current agent run. These are important to pass to future iteration so that agent knows what work it has already done.

Agent Inputs: The inputs to an agent is the key-value mapping and the required key is “intermediate_steps”.

Agent Outputs: The output is the next actions to take or the final response to send to the user. This can be typed as Union[AgentAction, List[AgentAction], AgentFinish].

Details about ReAct Agent

1. ReAct Agent

It allows LLMs to generate reasoning traces and task-specific actions, leveraging their synergy.
The framework involves an iterative process: given a question, LLMs first generate a reasoning step (thought) to decide how to tackle the problem and identify an action to take. Actions, like making API calls, are external tasks from an allowed set. LLMs then observe the results of these actions. If the answer is found, the process stops; otherwise, the LLM generates a new thought based on the observation and continues the cycle until the answer is obtained. Process is shown in the below image.

From the ReAct research paper, they outlines a simple Wikipedia web API with three actions for interactive information retrieval:

search[entity] — Returns the first 5 sentences from the entity’s wiki page or suggests the top-5 similar entities.
lookup[string] — Returns the next sentence on the page containing the string, simulating a browser’s Ctrl+F functionality.
finish[answer] — Completes the task with the provided answer.

Example: Question: What profession do MS Dhoni and Virat Kohli have in common?

Thought1: Search MS Dhoni and Virat Kohli, find their professions, and identify the common one.
Action1: search[MS Dhoni]
Observation1: Returns details about MS Dhoni being a professional cricketer.
Thought2: MS Dhoni is a professional cricketer. Next, search Virat Kohli and find his profession.
Action2: search[Virat Kohli]
Observation2: Returns details about Virat Kohli being a professional cricketer.
Thought3: Both MS Dhoni and Virat Kohli are professional cricketers.
Action3: finish[Professional cricketer]
Final Answer: Professional Cricketer.

LangChain includes an inbuilt agent called DocStore Explorer, which aligns with the functionality described in the original ReAct research paper for the Wikipedia use case.

The DocStore Explorer agent interacts with a document storage system (like Wikipedia) using two tools: the Search tool, which locates documents, and the Lookup tool, which retrieves terms from the most recently found document.

from langchain.agents.react.base import DocstoreExplorer
from langchain.docstore import Wikipedia
from langchain.agents import Tool

docstore = DocstoreExplorer(Wikipedia())

tools = [
    Tool(
        name="Search",
        func=docstore.search,
        description="useful for when you need to ask with search",
    ),
    Tool(
        name="Lookup",
        func=docstore.lookup,
        description="useful for when you need to ask with lookup",
    ),
]

import os
from langchain.chat_models import ChatOpenAI
llm = ChatOpenAI(openai_api_key=os.environ["OPEN_API_KEY"], model_name="gpt-4")

from langchain.agents import AgentType, initialize_agent
react = initialize_agent(tools, llm, agent=AgentType.REACT_DOCSTORE, verbose=True)

question = "What profession does MS Dhoni and Virat Kohli have in common?"
result = react.run(question)

"""
Response:

> Entering new AgentExecutor chain...
Thought: I need to search MS Dhoni and Virat Kohli, find their professions, then find the profession they have in common.
Action: Search[MS Dhoni]
Observation: Mahendra Singh Dhoni ( ; born 7 July 1981) is an Indian professional cricketer who plays as a right handed batter and a wicket-keeper. .......
Thought:MS Dhoni's profession is professional cricketer. I need to search Virat Kohli next and find his profession.
Action: Search[Virat Kohli]
Observation: Virat Kohli (Hindi pronunciation: [ʋɪˈɾɑːʈ ˈkoːɦli] ; born 5 November 1988) is an Indian international cricketer and the former captain ....... 
Thought:Virat Kohli's profession is also professional cricketer. Therefore, the profession MS Dhoni and Virat Kohli have in common is professional cricketer.
Action: Finish[Professional Cricketer]

> Finished chain.
"""

2. Zero Shot ReAct Agent

In “zero-shot-react-description” agent, “Zero-shot” means it operates solely on the current action without memory. It utilizes the ReAct framework to choose tools based only on their descriptions.
We will create the Calculator tool. When creating the application, we can use the pre-built tool or create the custom tool. For eg, we can create a new calculator tool from the existing llm_math chain.

from langchain.llms import OpenAI

os.environ["OPEN_API_KEY"]="" # OPENAPI KEY
llm=OpenAI(openai_api_key=os.environ["OPEN_API_KEY"],temperature=0)

from langchain.chains import LLMMathChain
from langchain.agents import Tool

llm_math = LLMMathChain(llm=llm)

# initialize the math tool
math_tool = Tool(
    name='Calculator Tool',
    func=llm_math.run,
    description='Useful, when to answer questions about maths.'
)

## when giving tools to LLM, we must pass as list of tools
tools = [math_tool]
tools[0].name, tools[0].description

"""
Response: 

('Calculator Tool', 'Useful, when to answer questions about maths.')
"""

from langchain.agents import initialize_agent

zero_shot_agent = initialize_agent(
    agent="zero-shot-react-description",
    tools=tools,
    llm=llm,
    verbose=True,
    max_iterations=3 # The number of thoughts, it can have about the questions and the tool.
)

zero_shot_agent("what is (10.5*2.5)^1.9?")

"""
Response: 

> Entering new AgentExecutor chain...
 I should use the calculator to solve this problem.
Action: Calculator
Action Input: (10.5*2.5)^1.9
Observation: Answer: 496.987750629
Thought: I now know the final answer.
Final Answer: 496.987750629

> Finished chain.

{'input': 'what is (10.5*2.5)^1.9?', 'output': '496.987750629'}

zero_shot_agent("what is the capital of India?")

"""
Response: 

"Error"
-- It gives error because, agent keeps trying to use a tool and as it is 
-- not the math problem, it throws the error
"""

To solve the above problem, we implement:

from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain

prompt = PromptTemplate(
    input_variables=["query"],
    template="{query}"
)

llm_chain = LLMChain(llm=llm, prompt=prompt)

# initialize the LLM tool
llm_tool = Tool(
    name='Language Model',
    func=llm_chain.run,
    description='use this tool for general purpose queries and logic'
)
tools.append(llm_tool)

# reinitialize the agent
zero_shot_agent = initialize_agent(
    agent="zero-shot-react-description",
    tools=tools,
    llm=llm,
    verbose=True,
    max_iterations=3
)

# Now it gives the correct result 
zero_shot_agent("what is the capital of India?")

"""
Response: 

{'input': 'what is the capital of India?', 'output': 'New Delhi'}
"""

3. Conversational ReAct

The zero-shot agent performs well but lacks conversational memory, which can be an issue for chatbots needing to remember previous interactions.
Fortunately, the conversational-react-description agent can retain interaction history, essentially functioning like the Zero Shot ReAct agent but with added conversational memory.
In the below example, let’s use agents to interact with a small sample database of stocks.

from sqlalchemy import MetaData
from sqlalchemy import Column, Integer, String, Table, Date, Float
from sqlalchemy import create_engine

metadata_obj = MetaData()
llm=OpenAI(openai_api_key=os.environ["OPEN_API_KEY"],temperature=0)

stocks = Table(
    "stocks",
    metadata_obj,
    Column("obs_id", Integer, primary_key=True),
    Column("stock_ticker", String(4), nullable=False),
    Column("price", Float, nullable=False),
    Column("date", Date, nullable=False),
)

engine = create_engine("sqlite:///:memory:")
metadata_obj.create_all(engine)

from datetime import datetime

observations = [
    [1, 'ABC', 200, datetime(2023, 1, 1)],
    [2, 'ABC', 208, datetime(2023, 1, 2)],
    [3, 'ABC', 232, datetime(2023, 1, 3)],
    [4, 'ABC', 225, datetime(2023, 1, 4)],
    [5, 'ABC', 226, datetime(2023, 1, 5)],
    [6, 'XYZ', 810, datetime(2023, 1, 1)],
    [7, 'XYZ', 803, datetime(2023, 1, 2)],
    [8, 'XYZ', 798, datetime(2023, 1, 3)],
    [9, 'XYZ', 795, datetime(2023, 1, 4)],
    [10, 'XYZ', 791, datetime(2023, 1, 5)],
]

from sqlalchemy import insert

def insert_obs(obs):
    stmt = insert(stocks).values(obs_id=obs[0], stock_ticker=obs[1], 
                                 price=obs[2], date=obs[3])
    with engine.begin() as conn:
        conn.execute(stmt)

for obs in observations:
    insert_obs(obs)

sql_tool = Tool(
    name='SQL Model',
    func=sql_chain.run,
    description='Use sqlite syntax to answer this query'
)

from langchain.memory import ConversationBufferMemory
memory = ConversationBufferMemory(memory_key="chat_history")

conversational_agent = initialize_agent(
    agent='conversational-react-description', 
    tools=[sql_tool], # To pass in the form of list
    llm=llm,
    verbose=True,
    max_iterations=3,
    memory=memory,
)

result = conversational_agent(
    "Please provide me the stock prices for ABC on January the 1st"
)

"""
Response: 

> Entering new AgentExecutor chain...
Thought: Do I need to use a tool? Yes
Action: SQL Model
Action Input: SELECT stock_price FROM stock_data WHERE stock_name = 'ABC' AND date = 'January 1st'

> Entering new SQLDatabaseChain chain...
SELECT stock_price FROM stock_data WHERE stock_name = 'ABC' AND date = 'January 1st'
SQLQuery:SELECT "price" FROM stocks WHERE "stock_ticker" = 'ABC' AND "date" = '2023-01-01'
SQLResult: [(200.0,)]
Answer:The stock price for 'ABC' on 'January 1st' was 200.0.
> Finished chain.

Observation: The stock price for 'ABC' on 'January 1st' was 200.0.
"""

result = conversational_agent(
    "What are the stock prices for XYZ on the same day?"
)

"""
Response: 

Thought: Do I need to use a tool? Yes
Action: SQL Model
Action Input: SELECT stock_price FROM stock_data WHERE stock_name = 'XYZ' AND date = 'January 1st'

> Entering new SQLDatabaseChain chain...
SELECT stock_price FROM stock_data WHERE stock_name = 'XYZ' AND date = 'January 1st'
SQLQuery:SELECT price FROM stocks WHERE stock_ticker = 'XYZ' AND date = '2023-01-01' LIMIT 5
SQLResult: [(810.0,)]
Answer:810.0
> Finished chain.

Observation: 810.0
Thought: Do I need to use a tool? No
AI: The stock price for XYZ on January 1st was $810.
"""

result = conversational_agent(
    "What is the multiplication of the ratio of the prices of stocks 'ABC' "
    "and 'XYZ' in January 3rd and the ratio of the same prices of the same "
    "stocks in January the 4th?"
)

"""
Response: 

Thought: Do I need to use a tool? Yes
Action: Calculator
Action Input: (200/810) * (210/820)
Observation: Answer: 0.06323396567299006
Thought: Do I need to use a tool? No
AI: The multiplication of the ratio of the prices of stocks 'ABC' and 'XYZ' in January 3rd and the ratio of the same prices of the same stocks in January the 4th is 0.06323396567299006.
"""

Advantages of ReAct Agent:

External Tool Integration: ReAct enables LLMs to interface with external tools, accessing additional information from various sources.
Human Alignment and Control: ReAct offers transparent decision-making, allowing human inspection and guidance, which enhances interpretability and trust.
Improved Performance: ReAct prompts LLMs to reason before answering, enhancing overall performance.

Limitations of ReAct Agent:

Dependence on Input Prompts: The model might rely on predefined prompts for possible actions, limiting its ability to perform unlisted actions.
Token Limitations: ReAct’s sequential reasoning and action process often requires more tokens, which may restrict the complexity of tasks in a single interaction.