Introducing Box tools for LlamaIndex

Image generated by deepai.org

Integrating enterprise content management systems with AI is essential for unlocking the potential of large language models.

Today, we’ll introduce Box tools for LlamaIndex — this powerful combination enables interaction between AI agents and Box content.

By leveraging these tools, developers can build intelligent applications capable of searching, extracting, and processing information from Box.

The Box tools for can be found in the LlamaIndex GitHub repo.

Box search

The Box search tool performs a search in a Box instance and returns LlamaIndex Documents to an AI agent.

Box search has plenty of parameters and options, so it’s necessary to define what type of search we want when we instantiate the tool, so that the AI agent can pass the search query.

The BoxSearchOptions class consists of:

class BoxSearchOptions:
    scope: Optional[SearchForContentScope] = None
    file_extensions: Optional[List[str]] = None
    created_at_range: Optional[List[str]] = None
    updated_at_range: Optional[List[str]] = None
    size_range: Optional[List[int]] = None
    owner_user_ids: Optional[List[str]] = None
    recent_updater_user_ids: Optional[List[str]] = None
    ancestor_folder_ids: Optional[List[str]] = None
    content_types: Optional[List[SearchForContentContentTypes]] = None
    limit: Optional[int] = None

To learn more about Box search, take a look at our guide, or this article:

Searching content in the Box Platform

Let’s see an example:

import openai
from box_sdk_gen import {
  DeveloperTokenConfig, 
  BoxDeveloperTokenAuth, 
  BoxClient,
}
from llama_index.agent.openai import OpenAIAgent
from llama_index.tools.box import BoxSearchToolSpec

openai.api_key = "your-openai-api-key"
BOX_DEV_TOKEN = "your_box_dev_token"

# Setup Box Client
config = DeveloperTokenConfig(BOX_DEV_TOKEN)
auth = BoxDeveloperTokenAuth(config)
box_client = BoxClient(auth)

# Import and initialize Box tool spec
box_tool_spec = BoxSearchToolSpec(box_client)

# Create an agent with the Box tool spec
agent = OpenAIAgent.from_tools(
    box_tool_spec.to_tool_list(),
    verbose=True,
)

answer = agent.chat("search all invoices")
print(answer)

Resulting in:

I found the following invoices:

1. **Invoice-A5555.txt**
   - Size: 150 bytes
   - Created By: RB Admin
   - Created At: 2024-04-30 06:22:18

2. **Invoice-Q2468.txt**
   - Size: 176 bytes
   - Created By: RB Admin
   - Created At: 2024-04-30 06:22:19

3. **Invoice-B1234.txt**
   - Size: 168 bytes
   - Created By: RB Admin
   - Created At: 2024-04-30 06:22:15

4. **Invoice-Q8888.txt**
   - Size: 164 bytes
   - Created By: RB Admin
   - Created At: 2024-04-30 06:22:14

5. **Invoice-C9876.txt**
   - Size: 189 bytes
   - Created By: RB Admin
   - Created At: 2024-04-30 06:22:17

These are the invoices found in the search. 
Let me know if you need more information or 
assistance with these invoices.

Box search by metadata

The Box search-by-metadata tool performs a metadata search in a Box instance and returns LlamaIndex Documents to an AI agent.

To execute a Box metadata search, you will need to specify the metadata template, the ancestor folder, and a parameterized query.

The BoxSearchByMetadataOptions class consists of:

class BoxSearchByMetadataOptions:
    from_: str
    ancestor_folder_id: str
    query: str
    limit: Optional[int] = None

To learn more about Box search, take a look at our guide, or this article:

Working with Box Metadata Queries

Here is an example:

import openai
from box_sdk_gen import {
  DeveloperTokenConfig, 
  BoxDeveloperTokenAuth, 
  BoxClient,
}
from llama_index.agent.openai import OpenAIAgent
from llama_index.tools.box import (
    BoxSearchByMetadataToolSpec,
    BoxSearchByMetadataOptions,
)

BOX_DEV_TOKEN = "your_box_dev_token"
openai.api_key = "your_openai_api_key"

# Setup Box Client
config = DeveloperTokenConfig(BOX_DEV_TOKEN)
auth = BoxDeveloperTokenAuth(config)
box_client = BoxClient(auth)


# Define your metadata search query
# This must be done when instantiating the tool 
# because the AI agent wont have a clue how to execute the search

# Parameters
from_ = "enterprise_" + "your_box_enterprise_id" + "." + "your_metadata_template_key"
ancestor_folder_id = "your_starting_folder_id"
query = "documentType = :docType "  # Your metadata query string
query_params = '{"docType": "Invoice"}'  # Your metadata query parameters

# Search options
options = BoxSearchByMetadataOptions(
    from_=from_,
    ancestor_folder_id=ancestor_folder_id,
    query=query,
)

box_tool = BoxSearchByMetadataToolSpec(box_client=box_client, options=options)

agent = OpenAIAgent.from_tools(
    box_tool.to_tool_list(),
    verbose=True,
)

answer = agent.chat(
    f"search all documents using the query_params as the key value pair of  {query_params} "
)
print(answer)

Note that the query parameters have been specified using a JSON string. This is to facilitate the AI agent passing the parameter into the tool.

Resulting in:

Added user message to memory: search all documents 
using the query_params as the key value pair of  {"docType": "Invoice"} 

=== Calling Function ===
Calling function: search with args: 
{"query_params":"{\"docType\": \"Invoice\"}"}
========================
I found the following documents with the 
query parameter "docType: Invoice":

1. Document ID: ee053400-e501-49d0-9e9f-e021bd86d9c6
   - Name: Invoice-B1234.txt
   - Size: 168 bytes
   - Created By: RB Admin
   - Created At: 2024-04-30 06:22:15

2. Document ID: 475cfee1-557a-4a3b-bc7c-18634d3a5d99
   - Name: Invoice-C9876.txt
   - Size: 189 bytes
   - Created By: RB Admin
   - Created At: 2024-04-30 06:22:17

3. Document ID: 9dbfd4ec-639f-4c7e-8045-645866e780a3
   - Name: Invoice-Q2468.txt
   - Size: 176 bytes
   - Created By: RB Admin
   - Created At: 2024-04-30 06:22:19

These are the documents matching the query parameter. 
Let me know if you need more information or assistance.

Box text extraction

Another possibility is to ask the AI agent to read a document stored in Box.

The Box extract tool retrieves the text representation of a document, and returns a LlamaIndex Document to the AI agent with the Document.text property containing the extracted text.

To learn more about Box text representation, take a look at our guide, or this article:

Unleashing the power of file representations with Box

Example:

import openai
from box_sdk_gen import {
  DeveloperTokenConfig, 
  BoxDeveloperTokenAuth, 
  BoxClient,
}
from llama_index.agent.openai import OpenAIAgent
from llama_index.tools.box import BoxTextExtractToolSpec

BOX_DEV_TOKEN = "your_box_dev_token"
openai.api_key = "your_openai_api_key"

# Setup Box Client
config = DeveloperTokenConfig(BOX_DEV_TOKEN)
auth = BoxDeveloperTokenAuth(config)
box_client = BoxClient(auth)

document_id = "your_document_id"

box_tool = BoxTextExtractToolSpec(box_client=box_client)

agent = OpenAIAgent.from_tools(
    box_tool.to_tool_list(),
    verbose=True,
)

answer = agent.chat(f"read document {document_id}")
print(answer)

Resulting in:

=== Calling Function ===
Calling function: extract with args: {"file_id":"1584056661506"}
Got output: Doc ID: da890d37-396a-4f0b-ad5a-1fb40f99782b
Text: Action Plan For  Teaching Economics 1   Long term Goal     To
teach Economics  by keeping in mind the 21st Century approaches and
skills 2   Short term Goal    To work on creativity and intellectual
curiosity  requirements of the students  To  develop interpersonal and
collaborative skills of the students  3
========================

I have extracted the text content from the document with ID 1584056661506. 
Here is a snippet of the text:

"Action Plan For Teaching Economics
1 Long term Goal
To teach Economics by keeping in mind the 21st Century approaches and skills
2 Short term Goal
To work on creativity and intellectual curiosity requirements of the students
To develop interpersonal and collaborative skills of the students"

Let me know if you need more information or assistance with this document.

What will you build?

Box tools for LlamaIndex offer a suite for interacting with Box content within an AI agent. By leveraging BoxSearchToolSpec, BoxSearchByMetadataToolSpec, and BoxTextExtractToolSpec, developers can search, filter, and extract information from Box documents.

In the context of a Retrieval-Augmented Generation (RAG) application, these tools enable the AI agent to access and process relevant information from Box, providing a foundation for generating informative and accurate responses.

For instance, an HR agent could utilize these tools to: search for employee documents based on specific criteria, extract key information, and generate summaries or reports.

In part 2 of this article, we’ll explore Box AI tools for LlamaIndex.

Stay tuned.

Thoughts? Comments? Feedback?

Drop us a line in our community forum.