Langchain and MRKL Agents Demystified
A large language model (LLM) can do a great job of writing a silly limerick or ripping off a Hollywood screenplay, but without access to affect the real world, GenAI is a curiosity at best, and a hallucinating copyright infringement liability at worst. The real power of GenAI is when you give it tools it can use, and the power to chain those tools together to accomplish complex tasks.
So, how exactly does this work? When I was learning this tech, I struggled to piece it all together from dense research papers, half-written library docs, marketing buzz, reading raw code, and trial-and-error. So I’ve brain-dumped this article in the hope I could help fill in the missing pieces for other AI newbies.
In this article we’ll explore the high-level concepts and go down into the details of how we can securely and reliably get LLMs to perform complex tasks with real-world data.
Basics: Tools and the MRKL Agent System
This process of chaining tools together is known as a MRKL (pronounced “miracle”) Agent System, or a “Modular Reasoning, Knowledge, and Language” system. Langchain is the most popular implementation, and at Clearwater, we use both standard Langchain and our own custom-built MRKL Agent System (from here on we’ll refer to an instance of this type of system as just an “Agent”).
A simple way to think of Agents is like this: LLMs can’t do math, but they can use a calculator.
An Over-Simplified Example: Give an LLM a Calculator
Here’s what happens if you just ask an LLM a math question:
There’s that fabrication stuff we talked about. 342,289,000 is in the ballpark, but close only counts in horseshoes and hand grenades. We need to do better.
Now let’s try that again, but with an Agent in the middle with access to a calculator:
Notes and Observations:
- We’ve previously defined a function in python code that acts as a calculator. It can parse the arithmetic expression and do the calculation, and we’ve registered it with the Agent.
- The Agent intercepts the user’s prompt BEFORE going to the LLM for an answer, and prepends some instructions on what tools are available and how to use them.
- The Agent sends the instructions AND the user’s prompt to the LLM, expecting the LLM to reason on the next steps, and get a specially structured response that traditional code can parse and execute.
- Note the interplay between “structured” and “unstructured” algorithms and data. We consider user prompts and LLM reasoning to be “unstructured,” but we’re asking the LLM to return a structured response that our structured traditional software can take action on.
- This requires multiple calls to an LLM, so it’s slower and more expensive. But LLMs are getting faster and cheaper (and better) at an incredible rate.
- Yes, this is vastly simplified. Stand by for more details.
Deep Dive: Fetch Balance Sheet and Show Top Positions
I hope you didn’t complain too much about how oversimplified that last example was, because check your diving gear — we’re going for a swim!
In this example, we’ll examine the entire process and the exact set of prompts sent in multiple steps to the LLM, using Langchain. We’ll set up three tools:
- report_data_exporter_s3: Fetch financial data as CSV file and store in S3.
- csv_file_sorter: Sort the CSV file by a column and show top 10. Read input from S3 and write output to S3.
- csv_file_viewer: Print the CSV file as a markdown table and show it to the user.
Each of these tools have been written as Python functions, and registered with Langchain as what’s known as a StructuredTool, including a description of what it is and how to use it, as well as a Pydantic object that serves as the input schema.
Ready? Let’s begin with a user typing in a request:
User: Show me the top 10 positions by Market Value for account 6. Use report -3 for date 2024-06-06
ENTER Agent Loop
At this point, we enter the Agent “loop.” This will result in multiple calls to the LLM, and at each step the LLM decides which action to take next until we reach a final answer. This consists of an initial instruction prompt, which:
- Informs the LLM of what tools are available, what they do, and what input params they require
- Instructs the LLM on how to answer: either use a tool with inputs or respond Final Answer
- Enters a loop:
- Asks LLM for an action
- Executes the Tool action and records results in a “scratch-pad”
- Repeats the loop, each time appending more response “thoughts” and more tool result “observations” in the “scratch-pad”
- Exits the loop once the LLM has responded with a Final Answer or the iteration limit has been reached (to avoid infinite loops)
Agent Loop 1
Instructions Prompt: (generated by Agent)
Respond to the human as helpfully and accurately as possible. You have access to the following tools:
report_data_exporter_s3: report_data_exporter_s3(**kwargs) -> str - This tool will fetch report data for a given pdid (Page Definition Id), Account Id, and Report Date.
The response data will be saved to s3 in CSV format, and a filename will be returned where the data can be found when needed.
action_input should be in the form of {"pdid": int, "account_id": int}
An example of action_input is {"pdid": -40, "account_id": 4}, args: {'pdid': {'title': 'Pdid', 'type': 'integer'}, 'account_id': {'title': 'Account Id', 'type': 'integer'}}
csv_file_sorter: csv_file_sorter(**kwargs) -> str - Use this tool to sort and page a csv file. Takes as input the csv filename and sorting columns/directions.
Offset/limit may be used for paging or viewing topN results, or omitted to simply sort a dataset.
A simple sort example of action input is {
"csv_filename": "abcd1234.csv",
"sort": [
{
"column": "Market Value"
"direction": "ASC"
},
}
A more complex example is {
"csv_filename": "abcd1234.csv",
"sort": [
{
"column": "Market Value"
"direction": "ASC"
},
{
"column": "Yield ",
"direction": "DESC",
}
]
"offset": 0,
"limit": 10
}, args: {'csv_filename': {'title': 'Csv Filename', 'type': 'string'}, 'sort': {'title': 'Sort', 'type': 'array', 'items': {'$ref': '#/definitions/CsvSortColumn'}}, 'offset': {'title': 'Offset', 'type': 'integer'}, 'limit': {'title': 'Limit', 'type': 'integer'}}
csv_file_viewer: csv_file_viewer(**kwargs) -> str - Use this tool to view a csv file. Takes as input the csv filename and an optional "limit" parameter to view only the first N rows.
Returns the contents of the file formatted as a markdown table.
An example of action input is {
"csv_filename": "abcd1234.csv",
"limit": 10
}, args: {'csv_filename': {'title': 'Csv Filename', 'type': 'string'}, 'limit': {'title': 'Limit', 'type': 'integer'}}
Use a json blob to specify a tool by providing an action key (tool name) and an action_input key (tool input).
Valid "action" values: "Final Answer" or report_data_exporter_s3, csv_file_sorter, csv_file_viewer
Provide only ONE action per $JSON_BLOB, as shown:
```
{
"action": $TOOL_NAME,
"action_input": $INPUT
}
```
Follow this format:
Question: input question to answer
Thought: consider previous and subsequent steps
Action:
```
$JSON_BLOB
```
Observation: action result
... (repeat Thought/Action/Observation N times)
Thought: I know what to respond
Action:
```
{
"action": "Final Answer",
"action_input": "Final response to human"
}
```
Begin! Reminder to ALWAYS respond with a valid json blob of a single action. Use tools if necessary. Respond directly if appropriate. Format is Action:```$JSON_BLOB```then Observation:.
Thought:User Prompt:
Show me the top 10 positions by Market Value for account 6. Use report -3 for date 2024-06-06
LLM Response:
Thought: To get the top 10 positions by Market Value for account 6, I need to first export the report data using the provided pdid and account id, then sort the CSV file by Market Value in descending order to get the top positions.
Action:
```
{
"action": "report_data_exporter_s3",
"action_input": {
"pdid": -3,
"account_id": 6,
"report_date": "2024-06-06"
}
}
```Tool Response:
The Agent executes the Tool as instructed (calls registered Python function with those params) and receives this response:
The CSV data can be found in the file: 9d8b6e932ad6.csv
Agent Loop 2
Since the LLM did not respond with “Final Answer,” the Agent records the action and its response in a “scratch-pad” and makes another call to the LLM:
Instructions Prompt:
... exactly the same as above - list of tools and instructions ...
User Prompt / scratchpad: (original question plus scratch-pad of prior responses — thoughts — and tool results — observations)
Show me the top 10 positions by Market Value for account 6. Use report -3 for date 2024-06-06
This was your previous work (but I haven't seen any of it! I only see what you return as final answer):
Thought: To get the top 10 positions by Market Value for account 6, I need to first export the report data using the provided pdid and account id, then sort the CSV file by Market Value in descending order to get the top positions.
Action:
```
{
"action": "report_data_exporter_s3",
"action_input": {
"pdid": -3,
"account_id": 6,
"report_date": "2024-06-06"
}
}
```
Observation: The CSV data can be found in the file: 9d8b6e932ad6.csv
Thought:LLM Response:
Now that I have the CSV filename, I need to sort the data by Market Value in descending order and limit the results to
the top 10 positions.
Action:
```
{
"action": "csv_file_sorter",
"action_input": {
"csv_filename": "9d8b6e932ad6.csv",
"sort": [
{
"column": "Market Value",
"direction": "DESC"
}
],
"limit": 10
}
}
```Tool Response:
The sorted data has been written to the file: 7f10686f2557.csv
Agent Loop 3
Instructions Prompt:
... exactly the same as above - list of tools and instructions ...
User Prompt / scratchpad:
Show me the top 10 positions by Market Value for account 6. Use report -3 for date 2024-06-06
This was your previous work (but I haven't seen any of it! I only see what you return as final answer):
Thought: To get the top 10 positions by Market Value for account 6, I need to first export the report data using the provided pdid and account id, then sort the CSV file by Market Value in descending order to get the top positions.
Action:
```
{
"action": "report_data_exporter_s3",
"action_input": {
"pdid": -3,
"account_id": 6,
"report_date": "2024-06-06"
}
}
```
Observation: The CSV data can be found in the file: 9d8b6e932ad6.csv
Thought:Now that I have the CSV filename, I need to sort the data by Market Value in descending order and limit the results to the top 10 positions.
Action:
```
{
"action": "csv_file_sorter",
"action_input": {
"csv_filename": "9d8b6e932ad6.csv",
"sort": [
{
"column": "Market Value",
"direction": "DESC"
}
],
"limit": 10
}
}
```
Observation: The sorted data has been written to the file: 7f10686f2557.csv
Thought:LLM Response:
Now that the data is sorted and the top 10 positions by Market Value are in the new CSV file, I need to view the contents of this file to provide the final answer.
Action:
```
{
"action": "csv_file_viewer",
"action_input": {
"csv_filename": "7f10686f2557.csv",
"limit": 10
}
}
```Tool Response:
Identifier|Description|Original Units|Current Units|Currency|Security Type|Final Maturity|Original Cost|Book Value|Interest/Dividend Due|Accrued Balance|Net Unrealized Gain/Loss|Market Price|Market Value|Market Value + Accrued
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CCYUSD|Cash|233,697,930.91|233,697,930.91|USD|CASH|05/07/2024|233,697,930.91|233,697,930.91|0.00|0.00|0.00|1.0|233697930.91|233,697,930.91
665278107|NORTHERN INSTITUTIONAL GOVT ASSTS|18,214,379.71|18,214,379.71|USD|MMFUND|05/07/2024|18,214,379.71|18,214,379.71|0.00|0.00|0.00|1.0|18214379.71|18,214,379.71
31326LRL1|FH 2B5891|25,000,000.00|2,229,819.75|USD|FHLMC|02/01/2047|2,288,413.37|2,287,523.90|22,608.97|2,636.14|42,586.76|104.4977|2330110.67|2,332,746.81
3140FCR59|FN BD5007|18,331,661.00|1,515,936.89|USD|FNMA|02/01/2047|1,544,405.80|1,543,972.04|7,569.58|1,766.23|26,215.83|103.5787|1570187.87|1,571,954.11
202795HK9|COMMONWEALTH EDISON CO|500,000.00|500,000.00|USD|CORP|03/15/2036|479,071.07|479,672.12|0.00|4,343.06|255,251.88|146.9848|734924.0|739,267.06
31347AJ47|FH 840283|12,000,000.00|691,027.32|USD|FHLMC|04/01/2046|704,433.67|704,222.50|6,783.00|791.69|5,504.44|102.7061|709726.93|710,518.62
745220FU2|PUERTO RICO COMWLTH INFRASTRUCTURE FING AUTH SPL T|1,000,000.00|1,000,000.00|USD|MUNI|07/01/2028|697,630.00|697,630.00|0.00|0.00|0.00|69.763|697630.0|697,630.00
925387AE2|VESSEL MANAGEMENT SERVIC|516,997.54|516,997.54|USD|CORP|04/16/2035|516,997.54|516,997.54|0.00|1,619.21|136,095.47|126.3242|653093.01|654,712.21
85749R479|SS INST INV:LIQ RSVS ADM|450,000.00|450,000.00|USD|MMFUND|05/07/2024|449,955.00|449,955.00|0.00|442.00|0.00|0.9999|449955.0|450,397.00
33611N101|FIRST PUERTO RICO TAX-EXEMPT TARGET MATURITY FUND|100,000.00|100,000.00|USD|Open-End Fund||272,000.00|272,000.00|0.00|0.00|0.00|2.72|272000.0|272,000.00Agent Loop 4:
Instructions Prompt:
... exactly the same as above - list of tools and instructions ...
User Prompt:
Show me the top 10 positions by Market Value for account 6. Use report -3 for date 2024-06-06
This was your previous work (but I haven't seen any of it! I only see what you return as final answer):
Thought: To get the top 10 positions by Market Value for account 6, I need to first export the report data using the provided pdid and account id, then sort the CSV file by Market Value in descending order to get the top positions.
Action:
```
{
"action": "report_data_exporter_s3",
"action_input": {
"pdid": -3,
"account_id": 6,
"report_date": "2024-06-06"
}
}
```
Observation: The CSV data can be found in the file: 9d8b6e932ad6.csv
Thought:Now that I have the CSV filename, I need to sort the data by Market Value in descending order and limit the results to the top 10 positions.
Action:
```
{
"action": "csv_file_sorter",
"action_input": {
"csv_filename": "9d8b6e932ad6.csv",
"sort": [
{
"column": "Market Value",
"direction": "DESC"
}
],
"limit": 10
}
}
```
Observation: The sorted data has been written to the file: 7f10686f2557.csv
Thought:Now that the data is sorted and the top 10 positions by Market Value are in the new CSV file, I need to view the contents of this file to provide the final answer.
Action:
```
{
"action": "csv_file_viewer",
"action_input": {
"csv_filename": "7f10686f2557.csv",
"limit": 10
}
}
```
Observation: Identifier|Description|Original Units|Current Units|Currency|Security Type|Final Maturity|Original Cost|Book Value|Interest/Dividend Due|Accrued Balance|Net Unrealized Gain/Loss|Market Price|Market Value|Market Value + Accrued
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CCYUSD|Cash|233,697,930.91|233,697,930.91|USD|CASH|05/07/2024|233,697,930.91|233,697,930.91|0.00|0.00|0.00|1.0|233697930.91|233,697,930.91
665278107|NORTHERN INSTITUTIONAL GOVT ASSTS|18,214,379.71|18,214,379.71|USD|MMFUND|05/07/2024|18,214,379.71|18,214,379.71|0.00|0.00|0.00|1.0|18214379.71|18,214,379.71
31326LRL1|FH 2B5891|25,000,000.00|2,229,819.75|USD|FHLMC|02/01/2047|2,288,413.37|2,287,523.90|22,608.97|2,636.14|42,586.76|104.4977|2330110.67|2,332,746.81
3140FCR59|FN BD5007|18,331,661.00|1,515,936.89|USD|FNMA|02/01/2047|1,544,405.80|1,543,972.04|7,569.58|1,766.23|26,215.83|103.5787|1570187.87|1,571,954.11
202795HK9|COMMONWEALTH EDISON CO|500,000.00|500,000.00|USD|CORP|03/15/2036|479,071.07|479,672.12|0.00|4,343.06|255,251.88|146.9848|734924.0|739,267.06
31347AJ47|FH 840283|12,000,000.00|691,027.32|USD|FHLMC|04/01/2046|704,433.67|704,222.50|6,783.00|791.69|5,504.44|102.7061|709726.93|710,518.62
745220FU2|PUERTO RICO COMWLTH INFRASTRUCTURE FING AUTH SPL T|1,000,000.00|1,000,000.00|USD|MUNI|07/01/2028|697,630.00|697,630.00|0.00|0.00|0.00|69.763|697630.0|697,630.00
925387AE2|VESSEL MANAGEMENT SERVIC|516,997.54|516,997.54|USD|CORP|04/16/2035|516,997.54|516,997.54|0.00|1,619.21|136,095.47|126.3242|653093.01|654,712.21
85749R479|SS INST INV:LIQ RSVS ADM|450,000.00|450,000.00|USD|MMFUND|05/07/2024|449,955.00|449,955.00|0.00|442.00|0.00|0.9999|449955.0|450,397.00
33611N101|FIRST PUERTO RICO TAX-EXEMPT TARGET MATURITY FUND|100,000.00|100,000.00|USD|Open-End Fund||272,000.00|272,000.00|0.00|0.00|0.00|2.72|272000.0|272,000.00
Thought:LLM Response:
I have the sorted data and can now provide the final answer to the user.
Action:
```
{
"action": "Final Answer",
"action_input": "Here are the top 10 positions by Market Value for account 6 as of 2024-06-06:
| Identifier | Description | Original Units | Current Units | Currency | Security Type | Final Maturity | Original Cost | Book Value | Interest/Dividend Due | Accrued Balance | Net Unrealized Gain/Loss | Market Price | Market Value | Market Value + Accrued |
|------------------|--------------------------------------------------|----------------|---------------|----------|---------------|----------------|---------------|------------|-----------------------|-----------------|-------------------------|--------------|--------------|------------------------|
| CCYUSD | Cash | 233,697,930.91 | 233,697,930.91 | USD | CASH | 05/07/2024 | 233,697,930.91 | 233,697,930.91 | 0.00 | 0.00
| 0.00 | 1.0 | 233,697,930.91 | 233,697,930.91 |
| 665278107 | NORTHERN INSTITUTIONAL GOVT ASSTS | 18,214,379.71 | 18,214,379.71 | USD | MMFUND | 05/07/2024 | 18,214,379.71 | 18,214,379.71 | 0.00 | 0.00
| 0.00 | 1.0 | 18,214,379.71 | 18,214,379.71 |
| 31326LRL1 | FH 2B5891 | 25,000,000.00 | 2,229,819.75 | USD | FHLMC | 02/01/2047 | 2,288,413.37 | 2,287,523.90 | 22,608.97 | 2,636.14
| 42,586.76 | 104.4977 | 2,330,110.67 | 2,332,746.81 |
| 3140FCR59 | FN BD5007 | 18,331,661.00 | 1,515,936.89 | USD | FNMA | 02/01/2047 | 1,544,405.80 | 1,543,972.04 | 7,569.58 | 1,766.23
| 26,215.83 | 103.5787 | 1,570,187.87 | 1,571,954.11 |
| 202795HK9 | COMMONWEALTH EDISON CO | 500,000.00 | 500,000.00 | USD | CORP | 03/15/2036 | 479,071.07 | 479,672.12 | 0.00 | 4,343.06
| 255,251.88 | 146.9848 | 734,924.00 | 739,267.06 |
| 31347AJ47 | FH 840283 | 12,000,000.00 | 691,027.32 | USD | FHLMC | 04/01/2046 | 704,433.67 | 704,222.50 | 6,783.00 | 791.69
| 5,504.44 | 102.7061 | 709,726.93 | 710,518.62 |
| 745220FU2 | PUERTO RICO COMWLTH INFRASTRUCTURE FING AUTH SPL T | 1,000,000.00 | 1,000,000.00 | USD | MUNI | 07/01/2028 | 697,630.00 | 697,630.00 | 0.00 | 0.00
| 0.00 | 69.763 | 697,630.00 | 697,630.00 |
| 925387AE2 | VESSEL MANAGEMENT SERVIC | 516,997.54 | 516,997.54 | USD | CORP | 04/16/2035 | 516,997.54 | 516,997.54 | 0.00 | 1,619.21
| 136,095.47 | 126.3242 | 653,093.01 | 654,712.21 |
| 85749R479 | SS INST INV:LIQ RSVS ADM | 450,000.00 | 450,000.00 | USD | MMFUND | 05/07/2024 | 449,955.00 | 449,955.00 | 0.00 | 442.00
| 0.00 | 0.9999 | 449,955.00 | 450,397.00 |
| 33611N101 | FIRST PUERTO RICO TAX-EXEMPT TARGET MATURITY FUND | 100,000.00 | 100,000.00 | USD | Open-End Fund | | 272,000.00 | 272,000.00 | 0.00 | 0.00
| 0.00 | 2.72 | 272,000.00 | 272,000.00 |"
}
```Note: The response contains the action “Final Answer,” so the Agent exits the loop, and returns the answer to the user.
EXIT Agent Loop
Now the Agent has completed its dirty work behind the scenes, and the user only sees the final answer as the response:
Assistant:
Here are the top 10 positions by Market Value for account 6 as of 2024-06-06:
| Identifier | Description | Original Units | Current Units | Currency | Security Type | Final Maturity | Original Cost | Book Value | Interest/Dividend Due | Accrued Balance | Net Unrealized Gain/Loss | Market Price | Market Value | Market Value + Accrued |
|------------------|--------------------------------------------------|----------------|---------------|----------|---------------|----------------|---------------|------------|-----------------------|-----------------|-------------------------|--------------|--------------|------------------------|
| CCYUSD | Cash | 233,697,930.91 | 233,697,930.91 | USD | CASH | 05/07/2024 | 233,697,930.91 | 233,697,930.91 | 0.00 | 0.00 | 0.00 | 1.0 | 233,697,930.91 | 233,697,930.91 |
| 665278107 | NORTHERN INSTITUTIONAL GOVT ASSTS | 18,214,379.71 | 18,214,379.71 | USD | MMFUND | 05/07/2024 | 18,214,379.71 | 18,214,379.71 | 0.00 | 0.00 | 0.00 | 1.0 | 18,214,379.71 | 18,214,379.71 |
| 31326LRL1 | FH 2B5891 | 25,000,000.00 | 2,229,819.75 | USD | FHLMC | 02/01/2047 | 2,288,413.37 | 2,287,523.90 | 22,608.97 | 2,636.14 | 42,586.76 | 104.4977 | 2,330,110.67 | 2,332,746.81 |
| 3140FCR59 | FN BD5007 | 18,331,661.00 | 1,515,936.89 | USD | FNMA | 02/01/2047 | 1,544,405.80 | 1,543,972.04 | 7,569.58 | 1,766.23 | 26,215.83 | 103.5787 | 1,570,187.87 | 1,571,954.11 |
| 202795HK9 | COMMONWEALTH EDISON CO | 500,000.00 | 500,000.00 | USD | CORP | 03/15/2036 | 479,071.07 | 479,672.12 | 0.00 | 4,343.06 | 255,251.88 | 146.9848 | 734,924.00 | 739,267.06 |
| 31347AJ47 | FH 840283 | 12,000,000.00 | 691,027.32 | USD | FHLMC | 04/01/2046 | 704,433.67 | 704,222.50 | 6,783.00 | 791.69 | 5,504.44 | 102.7061 | 709,726.93 | 710,518.62 |
| 745220FU2 | PUERTO RICO COMWLTH INFRASTRUCTURE FING AUTH SPL T | 1,000,000.00 | 1,000,000.00 | USD | MUNI | 07/01/2028 | 697,630.00 | 697,630.00 | 0.00 | 0.00 | 0.00 | 69.763 | 697,630.00 | 697,630.00 |
| 925387AE2 | VESSEL MANAGEMENT SERVIC | 516,997.54 | 516,997.54 | USD | CORP | 04/16/2035 | 516,997.54 | 516,997.54 | 0.00 | 1,619.21 | 136,095.47 | 126.3242 | 653,093.01 | 654,712.21 |
| 85749R479 | SS INST INV:LIQ RSVS ADM | 450,000.00 | 450,000.00 | USD | MMFUND | 05/07/2024 | 449,955.00 | 449,955.00 | 0.00 | 442.00 | 0.00 | 0.9999 | 449,955.00 | 450,397.00 |
| 33611N101 | FIRST PUERTO RICO TAX-EXEMPT TARGET MATURITY FUND | 100,000.00 | 100,000.00 | USD | Open-End Fund | | 272,000.00 | 272,000.00 | 0.00 | 0.00 | 0.00 | 2.72 | 272,000.00 | 272,000.00 |"Notes and Observations
Tool Descriptions
Let’s have a closer look at the Instructions Prompt from above. Here’s the excerpt from the csv_file_sorter Tool (formatted and annotated for clarity):
<<< PART ONE Generated Function Signature >>>
csv_file_sorter: csv_file_sorter(**kwargs) -> str
<<< PART TWO: Developer-written Description >>>
- Use this tool to sort and page a csv file. Takes as input the csv filename and sorting columns/directions.
Offset/limit may be used for paging or viewing topN results, or omitted to simply sort a dataset.
A simple sort example of action input is {
"csv_filename": "abcd1234.csv",
"sort": [
{
"column": "Market Value"
"direction": "ASC"
},
}
A more complex example is {
"csv_filename": "abcd1234.csv",
"sort": [
{
"column": "Market Value"
"direction": "ASC"
},
{
"column": "Yield ",
"direction": "DESC",
}
]
"offset": 0,
"limit": 10
},
<<< PART THREE: Generated Schema (formatted) >>>
args: {
'csv_filename': {'title': 'Csv Filename', 'type': 'string'},
'sort': {'title': 'Sort', 'type': 'array',
'items': {'$ref': '#/definitions/CsvSortColumn'}
},
'offset': {'title': 'Offset', 'type': 'integer'},
'limit': {'title': 'Limit', 'type': 'integer'}
}Part One: Generated Function Signature
This is generated by the Agent. It’s the method signature created by reflection.
Part Two: Developer-Written Description
This is the description of the tool the developer wrote. It is very important to be descriptive of what the tool can be used for and what it does. This is how the LLM can reason its way through the steps to chain the tools together.
Note the input example: This is also written by the developer. Sometimes the bare schema from Part Three isn’t enough on its own and the LLM makes mistakes. The sorter tool includes multiple examples, but it’s a good thing because the Langchain has a bug and didn’t fully serialize the complex schema object.
Part Three: Generated Schema
This is also generated by the Agent. It’s a schema generated from the Pydantic object registered as the args_schema for the Tool. This instructs the LLM how to format the JSON in the action_input so the Agent code can call the Tool function properly.
But if you look closely, you’ll see a subtle bug! It referenced CsvSortColumn type but never printed the full schema, so without the developer-written examples it would never work.
Here’s what the full Pydantic objects look like:
from pydantic.v1 import BaseModel as BaseModelV1
class CsvSortDirection(StrEnum):
ASCENDING = "ASC"
DESCENDING = "DESC"
class CsvSortColumn(BaseModelV1):
column: str
direction: CsvSortDirection
class CsvSorterToolSchema(BaseModelV1):
csv_filename: str
sort: list[CsvSortColumn]
offset: Optional[int] = None
limit: Optional[int] = NoneBest Practices with Tool Descriptions
It’s critical to be very clear about what the tool can do and how to use it. It’s best to think about an LLM as a person, and anything a person can’t figure out, an LLM certainly cannot (at least in this generation of LLMs). So put yourself in its shoes — if you had a complex task to accomplish and a set of random tools, what would you want to know?
There are four key parts to a tool description:
- What can it used for?
- What are the inputs?
- What are the outputs?
- What’s an example of how to use it?
The last part, an example of usage, should not be overlooked. Even if Langchain correctly generated a deep schema for all the nested input object types, sometimes that’s still not enough to get it to work right consistently. As a human, watching someone else do some task first is a lot easier than reading an instruction manual. Same goes for an AI.
Graceful Error Handling
So what happens if something goes wrong in the Agent? Let’s say you asked to sort by a column that doesn’t exist on the dataset. Do you throw an exception, dump a stack trace, return a generic error to the user, and dig through production logs after the user complains? Granted, sometimes catastrophic failure happens and that’s your only recourse, but in this brave new world sometimes we can use the Agent to “get back on the horse” and try again.
Handling Unrecoverable Errors
Let’s try to find the top 10 positions by Amortization Yield and see what happens
User: Show me the top 10 positions by Amortization Yield for account 6. Use report -3 for date 2024-06-06
ENTER Agent Loop
So far everything happens like before, except when the LLM gets to the sorting step.
LLM Response: sort that stuff by Amortization Yield [paraphrased]
Tool Response: langchain_core.tools.ToolException: Unable to find columns: Amortization Yield. Columns in this dataset are: Identifier,Description,Original Units,Current Units,Currency,Security Type,Final Maturity,Original Cost,Book Value,Interest/Dividend Due,Accrued Balance,Net Unrealized Gain/Loss,Market Price,Market Value,Market Value + Accrued.
Next Agent Loop
LLM Response: The column "Amortization Yield" is not present in the dataset. I need to sort by a different column that is available. Since the user requested "Amortization Yield" and it's not available, I should inform the user about this issue.
Action: { "action": "Final Answer", "action_input": "I'm sorry, but the column 'Amortization Yield' is not present in the dataset..." } [abridged]
Exit Agent Loop
Assistant: I'm sorry, but the column 'Amortization Yield' is not present in the dataset. The available columns are: Identifier, Description, Original Units, Current Units, Currency, Security Type, Final Maturity, Original Cost, Book Value, Interest/Dividend Due, Accrued Balance, Net Unrealized Gain/Loss, Market Price, Market Value, and Market Value + Accrued. Please let me know if you would like to sort by a different column or if there's anything else I can assist you with.
How did we do this? In Langchain, when you register a handle_tool_error function with your Tool, it will be called whenever a ToolException is raised. Then whatever string that function returns is sent as the Tool response. So in our Sort Tool, we first check if the dataset contains the column requested, and raise a ToolException with a helpful message.
Recovering and Continuing
So why did we include the list of available columns in our error message? Because we want to give that clever AI a chance to adapt and try again. Let’s put an incorrect column name in our prompt, but similar enough to one that does exist:
User: Show me the top 10 positions by Current Market Value for account 6. Use report -3 for date 2024-06-06
ENTER Agent Loop
So far everything happens like before, except when the LLM gets to the sorting step.
LLM Response: sort that stuff by Current Market Value [paraphrased]
Tool Response (observation): langchain_core.tools.ToolException: Unable to find columns: Current Market Value. Columns in this dataset are: Identifier,Description,Original Units,Current Units,Currency,Security Type,Final Maturity,Original Cost,Book Value,Interest/Dividend Due,Accrued Balance,Net Unrealized Gain/Loss,Market Price,Market Value,Market Value + Accrued.
Next Agent Loop
LLM Response:
```json
{
"action": "csv_file_sorter",
"action_input": {
"csv_filename": "54c9a1e3ffdf.csv",
"sort": [{"column": "Market Value", "direction": "DESC" }],
"limit": 10
}
}
```Hey, look at that! It knew exactly what to do — it just tried again with the correct equivalent column, and kept moving forward to get the right result.
Exit Agent Loop
Assistant: Here are the top 10 positions by Current Market Value for account 6 as of 2024-06-06 ... [abridged]
Notes and Observations
- Don’t tell the LLM what to do in your
ToolExceptionmessages - give the LLM enough information to be actionable, and let it make the decision on what to do next. - Just return the message from the Exception, not the full stack trace. There are various utilities in Python for this, but it’s annoying that it’s not built into the language.
- Our “Column Checker” is a shared utility that all of our data processing Tools use.
- This error handling pattern is nice and clean because anywhere deep in our tooling code we can just raise a helpful
ToolExceptionand the Agent either adapts and continues or responds helpfully to the user, and then the user can adapt and continue. - Always log your
ToolExceptionswith developer debug level info, even though your user already gets a nice error message. You’ll need this detailed info sooner or later.
Controlling Data Access
Security with AI is a very hot topic these days. We’re all aware of how LLMs fabricate or “hallucinate,” especially given ambiguous context, but they can also simply make mistakes, just like humans.
At Clearwater, we work with confidential investment portfolio data in a single-instance, multi-tenant system, so security is paramount. One of the first questions people ask us is about how we keep this data secure given the challenges with LLMs.
In the examples above, you’ll see we’re asking for portfolio data from account 6 — but what if we’re not allowed to see that account? There are essentially two cases we need to protect against:
- User asks for account data they aren’t allowed to see
- LLM makes a mistake and tries to access an account the user didn’t ask for
Fortunately, the solution to both cases is the same: never trust the LLM with data access!
Separate User Data from Agent Decisions
Given the critical importance of security, we recognized early on the need for a robust solution. So far, we’ve had very good results with the LLM doing the right thing, but by the nature of these things we know we cannot rely on it to be the gatekeeper.
The solution we developed is simple and reliable, keeping security concerns strictly in traditional code and algorithms:
- Each request to the Agent maintains the user identity data completely separate from any data going to/from the LLM.
- Any time the Agent executes a Tool, that Tool can validate data access with the original user identity.
- If access validation fails, throw a
ToolExceptionstating something like “User cannot access account 6.”
With this in place, we’re not only guaranteed not to leak unauthorized data, but we can also leverage the graceful error handling we learned about above, to either:
- Allow the LLM to try again, for example if it made a mistake confusing an account id with a security id, it can realize its error and do it correctly next iteration.
- Respond to the user exactly what account it tried to access and that they don’t have permission, and let them fix any typos or provide more clarity of what they wanted.
Respond with Context
One final thing to consider: what if the LLM accessed the wrong account, but the user actually does have access to it? In this case, we didn’t leak unauthorized data, we just returned the wrong data. It’s much less serious than unauthorized data access but still could cause the user to make a bad decision.
This is why it’s always best practice to respond back with something like “here’s the data for account 6,” or include the account id directly in the report data by the Tool that retrieved it, to make sure the user can confirm they’re getting what they asked for.
Confirmation Step
In cases where data access could take a long time, or you’re using the AI to [gasp] change some data, you’ll probably want the user to confirm what the AI is about to do. More detail on this will be the topic of a future article, but this is definitely something to keep in mind for advanced Agent workflows, both in terms of security as well as elegant UX.
Conclusion
Thanks for reading! I hope this was helpful. I’ve never been good at writing concluding paragraphs, so let’s ask my good buddy Claude3-Opus (who is usually pretty awesome but needs a bit more fine-tuning on where puns rank in the hierarchy of comedy).
Me: Hey Robot, can you help me write a short conclusion paragraph for the following tech blog post? Add a little humor. [copy/paste]
Claude: In conclusion, MRKL Agents and Langchain provide powerful tools for creating intelligent systems that can reason, access knowledge, and interact using natural language. By carefully designing tool descriptions, error handling, and secure data access patterns, developers can build robust and reliable AI agents capable of performing complex tasks with real-world data. As AI continues to advance, MRKL Agent architectures will play a crucial role in creating intelligent applications that adapt to user needs and provide valuable insights. So, the next time you’re impressed by an AI agent’s capabilities, just remember — it’s not a miracle, it’s a MRKL! 😄
About the Author
Phil Norton is a Principal Software Engineer in GenAI at Clearwater Analytics with a background in embedded systems, middleware, web development, management, and whatever else needs to be done at the time.
